{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 多层感知机"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上一章咱们学习了浅层线性神经网络，用它解决了简单的线性回归和多分类任务，想必同学们对神经网络的妙用已经开始有了一些体会，对pytorch的上手使用也有了初步的感性认识。本章我们将开始学习更加复杂的深层神经网络。首先，就是最简单的多层感知机（Multilayer Perceptron，MLP）了。它是一种非常流行的神经网络模型，包含至少一个隐藏层和输出层。先来看看和浅层神经网络相比，它的好处，或者说为什么要用它？\n",
    "\n",
    "一般来说，使用多层感知机的原因包括：\n",
    "\n",
    "1. 更好的解决非线性问题。多层感知机可以通过训练权值参数来学习非线性函数，这对于解决很多实际问题是很有用的。咱们前面学习的浅层神经网络只能解决线性问题或者简单的非线性问题，遇到稍微复杂点的数据就束手无策了。\n",
    "\n",
    "2. 它有较高的准确率。多层感知机可以通过训练来获得较高的准确率，尤其是当数据集较大时。虽然和后面学的动辄几十、上百，乃至上千层的神经网络没法比，但是MLP是它们的基础。先学好它，后面的学习会更容易。\n",
    "\n",
    "3. 它具有较好的泛化能力。多层感知机可以很好地适应新的数据，这使得它对于未来的数据具有较好的泛化能力。\n",
    "\n",
    "4. 它的训练方式灵活。多层感知机可以使用不同的优化算法和损失函数来训练，这使得它可以应用于各种不同的场景。\n",
    "\n",
    "知道了MLP的这么多好处，先来看看它的构成。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1.1 隐藏层\n",
    "\n",
    "多层感知机（Multilayer Perceptron, MLP）是一种前馈神经网络，它由输入层、隐藏层和输出层组成。输入层接收输入数据，隐藏层负责处理数据，输出层输出处理后的结果。\n",
    "\n",
    "<img src=\"../images/5-1-1.webp\" width=\"50%\"></img>\n",
    "\n",
    "\n",
    "隐藏层是多层感知机中最重要的一层。它位于输入层和输出层之间，不能直接与输入和输出打交道。隐藏层的作用是学习数据的高级表示，从而提高模型的准确度。\n",
    "\n",
    "隐藏层中的神经元接收输入层的输入数据，并使用权值和阈值进行线性组合，然后通过激活函数进行非线性转换。激活函数的作用是使得模型能够拟合非线性数据。常用的激活函数包括 sigmoid 函数、tanh 函数和 ReLU 函数。\n",
    "\n",
    "隐藏层的数量和大小对模型的性能有很大影响。如果隐藏层数量和大小均较小，则模型的表示能力较弱，可能无法准确地拟合训练数据。如果隐藏层数量和大小均较大，则模型的表示能力较强，但容易过拟合，即模型在训练数据上表现良好，但在测试数据上表现不佳。因此，在设计多层感知机时，需要考虑隐藏层的数量和大小，以达到较好的模型性能。\n",
    "\n",
    "此外，隐藏层中的神经元也可以使用不同的激活函数，这样可以让模型具有更强的表示能力。在实际应用中，我们可以使用超参数搜索来找到最优的隐藏层结构。\n",
    "\n",
    "在多层感知机中，隐藏层的数学表示为：\n",
    "\n",
    "$$h = f(W_hh_{prev} + b_h)$$\n",
    "\n",
    "其中，$h$ 是隐藏层的输出，$h_{prev}$ 是上一层（通常是输入层）的输出，$W_h$ 是隐藏层的权值矩阵，$b_h$ 是隐藏层的偏置向量，$f(\\cdot)$ 是激活函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1.2 激活函数\n",
    "\n",
    "激活函数是神经网络中非常重要的组成部分，它起到了非线性转换的作用。常见的激活函数有 sigmoid 函数、tanh 函数、ReLU 函数和 softmax 函数。\n",
    "\n",
    "**1. Sigmoid 函数**\n",
    "\n",
    "Sigmoid 函数是一种 S 形曲线，其表达式为：\n",
    "\n",
    "$$f(x) = \\frac{1}{1 + e^{-x}}$$\n",
    "\n",
    "在 PyTorch 中，可以使用 torch.nn 模块中的函数来实现上述激活函数并可视化它的图像。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAj/klEQVR4nO3deXxV1bn/8c8DIQkhCWMIY5hBRgUjoLYOdYJq0Q5aEac60Nurt9rawanaWu+t2l47iXWodR6KrQOtKNoW/FkVmWcZwhQIQxJCSELI/Pz+OAdvjEEOcJKdnPN9v1555Zx9Vs5+diDfrKy991rm7oiISOvXJugCREQkOhToIiIxQoEuIhIjFOgiIjFCgS4iEiMU6CIiMUKBLs3CzKaZ2dstbb9mNs/MrmvOmo6Ema02szOCrkNaBwW6RI2ZfcHMPjCzfWZWZGbvm9lJAO7+vLuf29w1Hct+zeynZlZtZmX1Pn4U7Rrr7e8pM7u3/jZ3H+nu85pqnxJbEoIuQGKDmaUDfwe+A8wEEoEvApVB1hUFf3b3y4MuQiQS6qFLtAwFcPcX3b3W3Q+4+9vuvgLAzK42s38fbGxm55rZunBv/mEze/fg0Ee47ftm9mszKzazTWZ2Snj7NjPLN7Or6r1XRzN7xswKzGyrmd1pZm0Osd9zzGxteL8PAXakBxruuT9X73l/M3MzSwg/n2dmPw8fQ6mZvW1m3eq1P/iXTHH4eK42s+nANOBH4b8E/hZuu8XMzg4/TjKz35jZjvDHb8wsKfzaGWa23cxuCX9/dprZt4702KR1U6BLtKwHas3saTObbGadD9UwHG5/AW4DugLrgFMaNJsArAi//gLwEnASMBi4HHjIzFLDbX8PdAQGAqcDVwKfCbPwfl8B7gS6ARuBU4/mYCNwWbiG7oT+WvlBuIZ+wJvhmjOAE4Bl7v4Y8DzwgLunuvtXGnnPO4CJ4a85HhgfPpaDehD6PvQGrgVmfN6/g8QeBbpEhbuXAF8AHHgcKDCzWWaW2UjzLwOr3f0Vd68BfgfsatBms7s/6e61wJ+BvsA97l7p7m8DVcBgM2sLXArc5u6l7r4F+F/gis/Z71/cvRr4TSP7beiScE/64Eevw34zQp509/XufoDQENQJ4e2XAf8I/yVT7e573H1ZhO85jdD3IN/dC4Cf8enjrA6/Xu3us4EyYFiE7y0xQIEuUePuH7v71e7eBxgF9CIUmg31ArbV+zoHtjdos7ve4wPhdg23pRLqabcDttZ7bSuhXmok+93WSLv6Zrp7p3ofOw7T/qD6vyjKw7VC6BfTxgjfo6FefPY46/+C2RP+BdnYfiUOKNClSbj7WuApQsHe0E6gz8EnZmb1nx+hQkI90371tmUBeYfYb98G++3bSLvD2Q+k1Hve4wi+dhsw6BCvHW7q0x189jgj/QUjcUCBLlFhZseFT8j1CT/vC0wF5jfS/A1gtJldFD6ReANHFoqfCA/JzAT+28zSwmPU3weea6T5G8BIM/taeL/fPcr9LgNOM7MsM+tI6FxApJ4HzjazS8wswcy6mtkJ4dd2EzoPcCgvAneaWUb4fMBdNH6cEqcU6BItpYROZH5kZvsJBfkq4JaGDd29ELgYeADYA4wAFnH0lzj+F6Fe8ybg34ROov7pc/Z7X3i/Q4D3j3Rn7v4OoXH9FcBiQpdrRvq1uYTG8m8Bigj9cjg+/PITwIjwWP1rjXz5vYS+TyuAlcCS8DYRAEwLXEjQwpcYbgemufvcoOsRaa3UQ5dAmNl5ZtYpfB317YSuB29seEZEIqRAl6CcTOhqj0LgK8BF4Uv8ROQoachFRCRGqIcuIhIjApucq1u3bt6/f/+gdi8i0iotXry40N0zGnstsEDv378/ixYtCmr3IiKtkpltPdRrGnIREYkRCnQRkRihQBcRiREKdBGRGHHYQDezP4VXQFl1iNfNzH5nZjlmtsLMxkW/TBEROZxIeuhPAZM+5/XJhCY5GgJMB/5w7GWJiMiROmygu/v/IzQr3KFcCDzjIfOBTmbWM1oFiohIZKJxHXpvPr3qy/bwtp0NG4YXwp0OkJWVFYVdi4i0HNW1dZRV1FBaUUNJRTWlFTWUVdZQWlHN/qpayitrKK+q5UvHdef4vp2ivv9mvbEovBDuYwDZ2dmaREZEWqSa2jqKD1Szd38Ve/ZXsXd/FUXlVRSVhT4f3F5yoJrSylCAl1ZUU1FdF9H7Z6QltdhAz+PTy3j1ofHlv0REAldRXcv2veVs33uA/JJKdpdUsKukgt3hx7tLKigsq6TuEF3OtKQEOndIpHOHRDqmJNKnSwrpyQmkJbcjNSmBtPDj0OcE0pJCjzskJZCS2Jb27drSpo01ybFFI9BnATea2UuEVqzZ5+6fGW4REWkuRfur2FhQRu6ecnKLytlWFPqcW1ROfulnF8bqnNKOzPRkMtOTGdEzncz0JLqlJdE5JZGu4fDu0iGRTintSEpoG8ARReawgW5mLwJnAN3MbDtwN6FV1nH3R4DZhJbUyiG0yvi3mqpYEZH69u6v4uOdJWzIL2NDfikbdpeRk1/Gnv1Vn7Qxg57pyfTtksLpQzPo2yWFrC4p9O7cnh7pyWSkJZHcruWG9JE4bKC7+9TDvO6EFvkVEWkye8oqWZm3j1V5+8KfS8gr/r81UdKSExiamcbZwzMZkpnKoIxU+nUNBXdL7lVHU2CzLYqIHIq7s2VPOQs272HB5r0s3FJEblH5J6/375rC2KxOXHlyP0b0SmdYZhoZaUmYNc3YdGuhQBeRFiF3Tznvrs/nw02hEC8sC411d+mQyEn9O3P5xCxG9+7EyN7ppCe3C7jalkmBLiKBqKiu5cONe3h3fQHvri9gc+F+AHp3as8Xh3TjpP5dGD+gM4MyUuO+5x0pBbqINJuyyhrmrs3nrdW7mLc2n/1VtSS3a8PJA7ty1cn9OH1Yd/p3TVGAHyUFuog0qcqaWuauzeeVJXnMW19AVU0d3VITmXJCb84bmcnEgV1j5iqToCnQRSTq3J2l24p5Zcl2/rZ8J/sOVJORlsS0CVlMHtWTE/t1pm0T3VwTzxToIhI1+ytreHVpHs9+uJV1u0tJbteG80b24Gvj+nDqoK4ktNUSDE1JgS4ix2xjQRnPfriVvy7eTmllDSN7pXPf10Zz/piepOmKlGajQBeRo7ZiezEPz93InDW7SGhjfHl0T648uT/jsjrpxGYAFOgickTcnY82FzFjbg7vbSgkLTmBG88czJUn9ycjLSno8uKaAl1EIrY0dy/3vbmWjzYX0S01kR9POo7LJ2ZpWKWFUKCLyGFtKijjV2+vY/bKXXRLTeTur4xg6vgsXW7YwijQReSQivZX8eA763hxwTaSEtpw89lDuO6LA0lNUnS0RPpXEZHPqKtzZi7axn1vraWsooZpE7L4ry8N0Rh5C6dAF5FPWb1jH3e+toqlucWMH9CFey8axdDMtKDLkggo0EUECE2W9as56/jT+5vpnJLIg5ccz1fH9tblh62IAl1EWL6tmO/PXMbGgv1cNiGLH593HB1TdOVKa6NAF4lj1bV1/P5fOcyYm0P3tCSeu3YCXxjSLeiy5Cgp0EXi1KaCMr770lJW5ZXwtbG9uXvKSDq2V6+8NVOgi8ShN1bs5Md/XUG7tsYjl49j0qieQZckUaBAF4kjVTV1/OLNj3ny/S2MzerEjMvG0atT+6DLkihRoIvEiZ37DnDD80tYklvMt07tz22Th5OYoOlsY4kCXSQOLN66l+nPLKKiupaHLhvLBWN6BV2SNAEFukiM+9vyHdzy8nJ6dkzmz98+mcHdU4MuSZqIAl0kRrk7D8/byC/nrOOk/p159IpsunRIDLosaUIKdJEYVFVTx+2vruQvi7dz0Qm9uP8bY0hK0MyIsU6BLhJjyqtq+Pazi3lvQyE3nz2Em84aotv344QCXSSGlFRUc82TC1mSu5cHvj6GS07qG3RJ0owU6CIxYk9ZJVc9uYB1u0r5/dRxnD9GNwvFGwW6SAzYXVLBtD9+xLaich67Mpszh3UPuiQJgAJdpJXbUXyASx+bT9H+Kp6+ZjwTB3YNuiQJSES3iZnZJDNbZ2Y5ZnZrI69nmdlcM1tqZivM7MvRL1VEGsovqeCyx+ezt7yK566boDCPc4cNdDNrC8wAJgMjgKlmNqJBszuBme4+FrgUeDjahYrIp+0pq2TaHz8iv7SSp741nhP6dgq6JAlYJD308UCOu29y9yrgJeDCBm0cSA8/7gjsiF6JItLQvvJqrnhiAblF5Txx1Umc2K9z0CVJCxBJoPcGttV7vj28rb6fApeb2XZgNvBfjb2RmU03s0VmtqigoOAoyhWR0opqrnxyATn5ZTx2ZTYnD9Iwi4REa6q1qcBT7t4H+DLwrJl95r3d/TF3z3b37IyMjCjtWiR+VNbUcv0zi1iVt4+HLhvL6UP1cyT/J5JAzwPq353QJ7ytvmuBmQDu/iGQDGgdK5EoqqtzfvDyCuZvKuJ/Lz6ec0f2CLokaWEiCfSFwBAzG2BmiYROes5q0CYXOAvAzIYTCnSNqYhE0f1z1vK35Tv48aTjuGhsw1FPkQgC3d1rgBuBOcDHhK5mWW1m95jZlHCzW4DrzWw58CJwtbt7UxUtEm+e+XALj767icsnZvEfpw8MuhxpoSK6scjdZxM62Vl/2131Hq8BTo1uaSIC8PbqXfx01mrOHp7Jz6aM0kRbckhaf0qkBVu+rZjvvrSU0X068fupY2nbRmEuh6ZAF2mh8ksqmP7sIrqlJvHEVdm0T9R85vL5NJeLSAtUWVPLt59bTMmBGl75z1PolpoUdEnSCijQRVoYd+fOV1exNLeYP0wbx/Ce6Yf/IhE05CLS4jz9wRZeXryd735pMJNHa05ziZwCXaQF+SCnkJ+/8THnjMjk5rOHBl2OtDIKdJEWYkfxAW54YQmDMjrw62+eQBtd0SJHSIEu0gJU19Zx4wtLqK51Hrn8RFKTdHpLjpz+14i0AA+8tZYlucU8dNlYBmakBl2OtFLqoYsE7O3Vu3j8vc1cdXI/LhjTK+hypBVToIsEKHdPObe8vJwxfTpy+/nDgy5HWjkFukhAKmtqueGFJQDMuGwcSQm6E1SOjcbQRQJy35trWZm3j8euOJG+XVKCLkdigHroIgGYty6fJ9/fwtWn9NdCFRI1CnSRZlZYVskPXl7BsMw0bp18XNDlSAzRkItIM3J3fvyXFZRUVPPcdeNJbqdxc4ke9dBFmtFzH+Xyz7X53Db5OI7roUm3JLoU6CLNJCe/lHv/vobTh2Zw9Sn9gy5HYpACXaQZVNXU8d0Xl9EhKYFfXjxGy8hJk9AYukgz+N0/N7BmZwmPX5lN97TkoMuRGKUeukgTW76tmD+8u5Gvj+vDOSMygy5HYpgCXaQJVVTXcsvLy8lITeKur4wIuhyJcRpyEWlCD76znpz8Mp6+Zjwd27cLuhyJceqhizSRxVuLePy9TUwdn8XpQzOCLkfigAJdpAkcqKrlBy+voFfH9tyhWRSlmWjIRaQJPDBnLZsL9/PC9RO0+pA0G/XQRaJsSe5envpgC1dM7Mcpg7oFXY7EEQW6SBRV1dRx619X0CM9mR9NGhZ0ORJn9LegSBQ9PC+H9bvL+NPV2aQl66oWaV7qoYtEyYbdpcyYm8OU43vxpeN0A5E0PwW6SBTU1jk//usKUpMSuFs3EElAIgp0M5tkZuvMLMfMbj1Em0vMbI2ZrTazF6JbpkjL9uyHW1iSW8xPLhhB19SkoMuROHXYMXQzawvMAM4BtgMLzWyWu6+p12YIcBtwqrvvNbPuTVWwSEuTV3yAB+as47ShGXx1bO+gy5E4FkkPfTyQ4+6b3L0KeAm4sEGb64EZ7r4XwN3zo1umSMvk7tzx6koA/uerozQtrgQqkkDvDWyr93x7eFt9Q4GhZva+mc03s0mNvZGZTTezRWa2qKCg4OgqFmlBXl+2g3nrCvjhecPo0zkl6HIkzkXrpGgCMAQ4A5gKPG5mnRo2cvfH3D3b3bMzMjS3hbRuxeVV/PzvazihbyeuPLl/0OWIRBToeUDfes/7hLfVtx2Y5e7V7r4ZWE8o4EVi1v1vraP4QDW/+Npo2rbRUIsEL5JAXwgMMbMBZpYIXArMatDmNUK9c8ysG6EhmE3RK1OkZVmSu5cXF+TyrVP6M7ynFnuWluGwge7uNcCNwBzgY2Cmu682s3vMbEq42Rxgj5mtAeYCP3T3PU1VtEiQamrruOPVVfRIT+bmc4YGXY7IJyK69d/dZwOzG2y7q95jB74f/hCJaU9/uJWPd5bwh2njNJOitCi6U1TkCOzaV8GDb6/jjGEZTBrVI+hyRD5FgS5yBH7+9zXU1Dn3TNE159LyKNBFIvTu+gLeWLmTG88cTFZXXXMuLY8CXSQCFdW13PX6KgZmdGD66QODLkekUTqjIxKBh+dtZOuecl64bgJJCW2DLkekUeqhixzGpoIyHpm3kQtP6MUpg7WknLRcCnSRz+Hu3PX6apLateGO84cHXY7I51Kgi3yOv63Yyb9zCvnhecPonpYcdDkin0uBLnIIJRXV/PzvaxjTpyPTJvQLuhyRw9JJUZFDePDt9RSWVfLEVdmafEtaBfXQRRqxKm8fz3y4hSsm9mNMn05BlyMSEQW6SAO1daFViLp0SOKWc4cFXY5IxBToIg28sCCX5dv38ZMLhtOxfbugyxGJmAJdpJ6C0koeeGstpw7uypTjewVdjsgRUaCL1PM/sz+msrqOey7U5FvS+ijQRcI+2FjIq0vz+PbpAxmUkRp0OSJHTIEuAlTV1PGT11aR1SWFG84cHHQ5IkdF16GLAI+/t4mNBft58lsnkdxOk29J66QeusS9bUXl/O6fG5g8qgdnDusedDkiR02BLnHN3bl71moS2hh3fWVE0OWIHBMFusS1Oat386+1+XzvnKH07Ng+6HJEjokCXeJWWWUNP/vbaob3TOfqU/oHXY7IMVOgS9z6zTvr2VVSwX9/dRQJbfWjIK2f/hdLXFq9Yx9PfrCFqeOzGJfVOehyRKJCgS5xp7bOuf3VVXROacePzzsu6HJEokaBLnHnxQW5LN9WzJ3nj6BjiibfktihQJe4UlBayf1vreWUQV258ARNviWxRYEuceXeN9ZQWV3Hzy/S5FsSexToEjf+vaGQ15ft4DtnDNLkWxKTFOgSFyqqa/nJ66vo3zWF75wxKOhyRJqEJueSuPDIuxvZXLifZ68dr8m3JGZF1EM3s0lmts7Mcszs1s9p93UzczPLjl6JIsdmc+F+Hp67kSnH9+KLQzKCLkekyRw20M2sLTADmAyMAKaa2WdmMTKzNOAm4KNoFylytNxDCz4ntWvDnRcMD7ockSYVSQ99PJDj7pvcvQp4CbiwkXY/B+4HKqJYn8gxeXnRdj7YuIdbJx9H97TkoMsRaVKRBHpvYFu959vD2z5hZuOAvu7+xue9kZlNN7NFZraooKDgiIsVORL5pRXc+8Yaxg/owtSTsoIuR6TJHfNVLmbWBngQuOVwbd39MXfPdvfsjAyNZUrT+ums1VTU1HHf10bTpo2uOZfYF0mg5wF96z3vE952UBowCphnZluAicAsnRiVIM1ZvYvZK3dx01lDGKhrziVORBLoC4EhZjbAzBKBS4FZB190933u3s3d+7t7f2A+MMXdFzVJxSKHse9ANT95bRXDe6Yz/bSBQZcj0mwOG+juXgPcCMwBPgZmuvtqM7vHzKY0dYEiR+q+N9dSWFbJ/V8fTTvNcy5xJKIbi9x9NjC7wba7DtH2jGMvS+TozN+0hxcX5DL9tIGM6dMp6HJEmpW6LxIzKqprue2VlWR1SeF7Zw8NuhyRZqdb/yVm/PafG9hcuJ/nr5tA+0Td3i/xRz10iQnLtxXz6LsbuSS7D6cO7hZ0OSKBUKBLq1dRXcstLy8nMz2ZOy/4zKwUInFDQy7S6j34znpy8st45prxpCdrSTmJX+qhS6u2aEsRj7+3icsmZHHaUN19LPFNgS6tVnlVDT94eTm9O7Xn9i9rJkURDblIq/XAW+vYsqecF6+fSGqS/iuLqIcurdIHGwt56oMtXH1Kf04e1DXockRaBAW6tDr7yqu5ZeZyBnTrwI8mDQu6HJEWQ3+nSqvi7tz+6koKSit55T9PISVR/4VFDlIPXVqVvy7J442VO/n+uUM1V4tIAwp0aTW27tnP3a+vYsKALnz7tEFBlyPS4ijQpVWorq3jppeW0baN8etvnkBbrUAk8hkagJRW4ff/ymHZtmJmXDaOXp3aB12OSIukHrq0eB9sLOShf23g6+P6cP6YnkGXI9JiKdClRcsvreC7Ly5jQLcO3HPhyKDLEWnRNOQiLVZtnXPTi8soq6zm+esm0EF3g4p8Lv2ESIv123+s58NNe/jVxcczrEda0OWItHgacpEW6d31Bfx+bg6XZPfhGyf2CbockVZBgS4tzs59B/jen5cxLDONn00ZFXQ5Iq2GAl1alIrqWv7z+SVUVtcyY9o4rQ0qcgQ0hi4thrtz52urWJpbzB+mjWNQRmrQJYm0KuqhS4vx1Adb+Mvi7Xz3rCFMHq3rzUWOlAJdWoT3cwq5942POXdEJjefNSTockRaJQW6BC53Tzk3vLCEQRkdePCbJ9BG87SIHBUFugSqrLKG659ZhDs8fmW2lpITOQb66ZHAVNfW8Z3nFpNTUMbT3xpPv64dgi5JpFVTD10C4e7c/spK3ttQyC++OpovDOkWdEkirZ4CXQLxm39s4OXF27nprCFcclLfoMsRiQkKdGl2Mxdu47f/3MDFJ/bh5rN1RYtItEQU6GY2yczWmVmOmd3ayOvfN7M1ZrbCzP5pZv2iX6rEgnnr8rnt1ZV8cUg3/udrozHTFS0i0XLYQDeztsAMYDIwAphqZiMaNFsKZLv7GOAvwAPRLlRavwWbi/jOc0sYlpnGw9PG0a6t/kAUiaZIfqLGAznuvsndq4CXgAvrN3D3ue5eHn46H9D0ePIpy7YVc81TC+nZKZmnrxlPWnK7oEsSiTmRBHpvYFu959vD2w7lWuDNxl4ws+lmtsjMFhUUFERepbRqq3fs48onPqJLh0ReuG4iGWlJQZckEpOi+jevmV0OZAO/bOx1d3/M3bPdPTsjIyOau5YWasPuUq54YgEdkhJ4/roJ9OiYHHRJIjErkhuL8oD615X1CW/7FDM7G7gDON3dK6NTnrRmmwv3M+2PH9G2jfHC9RPp2yUl6JJEYlokPfSFwBAzG2BmicClwKz6DcxsLPAoMMXd86NfprQ2a3eVcPEjH1JT5zx/3QQGdNNdoCJN7bCB7u41wI3AHOBjYKa7rzaze8xsSrjZL4FU4GUzW2Zmsw7xdhIHlm0r5puPziehjTHz2yczNFPrgYo0h4jmcnH32cDsBtvuqvf47CjXJa3U/E17uPaphXRNTeL56yZomEWkGWlyLomauevy+Y9nF5PVJYXnrptAZrpOgIo0JwW6RMVLC3K587VVHNczjWeumUCXDolBlyQSdxTockzq6pz756zl0Xc3cdrQDB66bCzpumlIJBAKdDlqB6pq+f7MZby5ahfTJmTxsykjSdDt/CKBUaDLUckvreD6ZxazYnsxd54/nGu/MEATbYkETIEuR2zB5iJufGEJpRU1PHr5iZw7skfQJYkICnQ5Au7OH9/bzH1vraVv5/Y8fc14hvdMD7osEQlToEtESiqq+eHLy5mzejeTRvbggYvH6OSnSAujQJfDWpq7l5v/vIztew9ovFykBVOgyyFV1dTxu39u4OF5OfRIT+al6RM5qX+XoMsSkUNQoEuj1u0q5Xt/XsaanSV848Q+3PWVERpiEWnhFOjyKVU1dfzx35v4zTsbSEtO4LErdBWLSGuhQJdPzN+0h5+8tooN+WVMGtmDe786im6pWl1IpLVQoAsFpZX8YvbHvLI0jz6d2/PEVdmcNTwz6LJE5Agp0ONYRXUtz364ld/9awMV1bXceOZgbjhzMO0T2wZdmogcBQV6HKqtc15dmseDb69jx74KThuawd1fGcGgjNSgSxORY6BAjyPuzr/W5vPLOetYu6uU0b078quLj+eUwd2CLk1EokCBHgdq65y3Vu1ixtwc1uwsoV/XFH4/dSznj+5Jmza6QUgkVijQY1hVTR2vLcvjkXkb2VS4n4HdOvDAN8Zw0Qm9SUzQNLcisUaBHoN27avghY+28sKCbRSWVTKyVzoPTxvHeSN70FY9cpGYpUCPEXV1zvzNe3j2w628vWY3de6cOaw7V53Sn9OGdNPcKyJxQIHeym0u3M+rS7bzytI8tu89QKeUdlz3hQFcPrEffbukBF2eiDQjBXortGtfBXNW7+L1ZXksyS2mjcEXhmTwg3OHMWlUD5Lb6TpykXikQG8ltu7Zz1urdvHW6l0szS0GYFhmGrdNPo6LxvYmMz052AJFJHAK9BaqorqW+Zv28O76At5dX8Cmgv0AjO7dkR+eN4zzRmYyuHtawFWKSEuiQG8hDlTVsnTbXhZu3svCLUUs3FJEZU0dSQltmDiwK5dP6Mc5IzI1Li4ih6RAD4C7s7ukkpV5+1i0pYgFW4pYlbeP6lrHDI7rkc7lE/tx+tAMxg/oojFxEYmIAr2JuTs79lWwKm/fJx8r80ooLKsEoF1b4/g+nbjuiwMZ378L4/p1pmN7LSQhIkdOgR4lNbV15BUfICe/jA35Zf/3eXcp+6tqAWjbxhjSPZXTh2Ywunc6o3p3ZFTvjuqBi0hUKNAjVFfn7C2vIq/4ALlF5eQWlbMt/Dm3qJwdxRXU1vkn7TPTkxjSPY2Ls/syuHsqI3ulM7xnusJbRJpM3Ad6VU0dxeVV7C2vJr+0gt0llewuqSC/JPy4tIL8kkrySyuorvVPfW3XDon07ZLC2L6dufD4FLK6pDCoeyqDu6dq2EREml1EgW5mk4DfAm2BP7r7fQ1eTwKeAU4E9gDfdPct0S21cbV1TlllTeijouZTj/dX1lBaWUNpRTXF5dXsDQf33v1V7C2vori8mrLKmkbfNz05gcz0ZDLTk5kwsAPd05LpnpZE787tyeqSQt8uKaQmxf3vQxFpQQ6bSGbWFpgBnANsBxaa2Sx3X1Ov2bXAXncfbGaXAvcD32yKgv+8MJdH391EaTi0D1TXRvR1ackJdE5JpHOHRLqmJjK4e2roeUo7OndIpHNKIhlpSWSmJ9E9LVmr9ohIqxNJF3M8kOPumwDM7CXgQqB+oF8I/DT8+C/AQ2Zm7v7pMYoo6NIhiRG90klLTiA1KYEOSaHPoeft6JDU9pPHqQfbJLYloa2mixWR2BZJoPcGttV7vh2YcKg27l5jZvuArkBh/UZmNh2YDpCVlXVUBZ8zIpNzRmgBYxGRhpq12+ruj7l7trtnZ2RkNOeuRURiXiSBngf0rfe8T3hbo23MLAHoSOjkqIiINJNIAn0hMMTMBphZInApMKtBm1nAVeHH3wD+1RTj5yIicmiHHUMPj4nfCMwhdNnin9x9tZndAyxy91nAE8CzZpYDFBEKfRERaUYRXUjt7rOB2Q223VXvcQVwcXRLExGRI6Fr+UREYoQCXUQkRijQRURihAV1MYqZFQBbA9n5selGgxum4kC8HXO8HS/omFuTfu7e6I08gQV6a2Vmi9w9O+g6mlO8HXO8HS/omGOFhlxERGKEAl1EJEYo0I/cY0EXEIB4O+Z4O17QMccEjaGLiMQI9dBFRGKEAl1EJEYo0I+Bmd1iZm5m3YKupSmZ2S/NbK2ZrTCzV82sU9A1NRUzm2Rm68wsx8xuDbqepmZmfc1srpmtMbPVZnZT0DU1FzNra2ZLzezvQdcSLQr0o2RmfYFzgdyga2kG7wCj3H0MsB64LeB6mkS99XMnAyOAqWY2ItiqmlwNcIu7jwAmAjfEwTEfdBPwcdBFRJMC/ej9GvgREPNnld39bXevCT+dT2iRk1j0yfq57l4FHFw/N2a5+053XxJ+XEoo4HoHW1XTM7M+wPnAH4OuJZoU6EfBzC4E8tx9edC1BOAa4M2gi2gija2fG/PhdpCZ9QfGAh8FXEpz+A2hDlldwHVEVUTzoccjM/sH0KORl+4Abic03BIzPu943f31cJs7CP2J/nxz1iZNz8xSgb8CN7t7SdD1NCUzuwDId/fFZnZGwOVElQL9ENz97Ma2m9loYACw3MwgNPywxMzGu/uuZiwxqg51vAeZ2dXABcBZMby8YCTr58YcM2tHKMyfd/dXgq6nGZwKTDGzLwPJQLqZPefulwdc1zHTjUXHyMy2ANnu3hpnbYuImU0CHgROd/eCoOtpKuEFztcDZxEK8oXAZe6+OtDCmpCFeiVPA0XufnPA5TS7cA/9B+5+QcClRIXG0CUSDwFpwDtmtszMHgm6oKYQPvF7cP3cj4GZsRzmYacCVwBfCv/bLgv3XKUVUg9dRCRGqIcuIhIjFOgiIjFCgS4iEiMU6CIiMUKBLiISIxToIiIxQoEuIhIj/j9N+IsDTNfbwAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "x = torch.linspace(-5, 5, 200)\n",
    "y = torch.sigmoid(x)\n",
    "\n",
    "plt.plot(x.numpy(), y.numpy())\n",
    "plt.title(\"Sigmoid Function\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sigmoid 函数的优点是输出值域范围为 (0,1)，适合二分类问题。缺点是当输入较大时，梯度较小，容易导致梯度消失问题。\n",
    "\n",
    "**2. Tanh 函数**\n",
    "\n",
    "Tanh 函数是一种双曲正切函数，其表达式为：\n",
    "\n",
    "$$f(x) = \\frac{e^x - e^{-x}}{e^x + e^{-x}}$$\n",
    "\n",
    "下面是它的pytorch实现和可视化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEICAYAAABS0fM3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAk/0lEQVR4nO3deZhU5Zn38e/dG82+NvsuKKAiaotGjVFERKNi1HGLCU7iOJMZM9knGt+JiYl5TTJvzMzESeKoE2c0otEkolFBUBMdBWkMIM3a7N30BgjN1lvV/f5Rp03RdLNVVZ+uqt/nuuqqc57znKq7oKt+dZY6j7k7IiKSvXLCLkBERMKlIBARyXIKAhGRLKcgEBHJcgoCEZEspyAQEclyCgLJamZ2u5m9HXYdx8rMPm5ma8OuQzKLgkDShpnti7tFzexg3PynO6iGza2ed5+ZDU3h87mZjWuZd/e33P2UVD2fZKe8sAsQOVbu3qNl2sw2A3e4+4IQSrk6pOcVSQltEUjaM7OpZvaume02s0oz+5mZFcQtdzP7OzNbH/R52Mys1WP8i5l9aGabzOyKE6hhs5lNj5v/jpk9GUyPDmqYbWZbzWyHmd0b1zfXzL5lZhvMbK+ZLTWzEWb2p6DL8mDL4yYzu9jMyuPWnWhmbwavq9TMrolb9qvgtf4heNzFZnbS8b42yXwKAskEEeArwADgY8ClwN+36nMVcA4wGbgRuDxu2bnA2mD9HwGPtQ6KJLkQOCWo79tmNjFo/ypwC3Al0Av4HHDA3S8Klp/h7j3c/Zn4BzOzfOBFYD4wEPgi8JSZxe86uhn4LtAXKAMeSMHrkjSnIJC05+5L3X2Ruze7+2bgl8AnWnV70N13u/tW4A1gStyyLe7+n+4eAZ4AhgCDjvCUvw++ge82s98fR6nfdfeD7r4cWA6cEbTfAfwfd1/rMcvdfecxPN55QI/gtTW6++vAS8RCpcXv3P09d28GnuLQ1y0C6BiBZAAzOxn4CVAMdCP2d720VbequOkDxD5AD1vm7geCjYH45a1de4LHCNqrYQSw4QQebyiwzd2jcW1bgGHH8JwiH9EWgWSCnwNrgPHu3gv4FpCKXTtHsp9YCLUYfBzrbgNOZN/9dmCEmcW/j0cCFSfwWJLFFASSCXoCdcA+M5sAfCGEGpYBN5tZvpkVAzccx7qPAt8zs/EWM9nM+gfLqoGx7ay3mNi3/H8Knvdi4Gpgzom8AMleCgLJBF8HbgX2Av8JPHPk7inxz8S+1X9I7ODsr49j3Z8AzxI76FsHPAZ0DZZ9B3giOB5xY/xK7t5I7IP/CmAH8B/AZ919zYm/DMlGpoFpRESym7YIRESynIJARCTLKQhERLKcgkBEJMul5Q/KBgwY4KNHjw67DBGRtLJ06dId7l7Uuj0tg2D06NGUlJSEXYaISFoxsy1ttWvXkIhIllMQiIhkOQWBiEiWUxCIiGQ5BYGISJZLShCY2eNmVmNmK9tZbmb2b2ZWZmYrzOysuGWzgyEE15vZ7GTUIyIixy5ZWwS/AmYeYfkVwPjgdiex68djZv2A+4gNFTgVuM/M+iapJhEROQZJ+R2Bu//JzEYfocss4L89dqnTRWbWx8yGABcDr7n7LgAze41YoDydjLpEJHNFo059c4SDjREamqNEok5TJEpz1GmOOM3RVtOH3Mdu0agTdced2D3gH82DE5v2YFk0GtwHbUDc+q36x13YOf4az4e2ezvt7awAzD5/NP17dEnwX+9QHfWDsmHERmFqUR60tdd+GDO7k9jWBCNHjkxNlSLSIZojUXbub6SmroEd+xrYc7CJuvom9hyI3dcdbI7d1zexr76Z+qYoB5si1H90i9IYiR79iTKExY23d82UYWkbBAlz90eARwCKi4s1iIJIJ1bfFKH8wwNs3nGALbsOsGXnfso/PEh1XT01exvYua+BaDvv4q75ufTqmkfvrvn0KsynT7cCuubnUpifQ9eCXLrk5VKYn9uqLYe8nBzyco28nBxyc4z8XAvu4+dzyMuxj/rlGOSYkWOGGcHNMDi0DSMnWJYTzGMc1ta6fzyL+zS3Q9rb7tOROioIKogN0N1ieNBWQWz3UHz7mx1Uk4gkwZ4DTSwv382qyjpWV9axansdG3fsJxL3Sd+zMI8RfbsxuHchpw/rzcCeXSjqVRi779mFPl3z6d01n56F+RTk6WTGjtZRQTAXuMvM5hA7MLzH3SvNbB7wg7gDxDOAezqoJhE5AbsPNPLepl0s2riLRRt3srqq7qPd2EN7FzJpaC9mnjaYk4p6MKp/N0b3706fbvmhfduVo0tKEJjZ08S+2Q8ws3JiZwLlA7j7L4CXgSuBMmKDbf91sGyXmX0PWBI81P0tB45FpPMo//AA80qrmVdaRcnmXUQduuTlcPaovnxl+skUj+rLpKG96NOtIOxS5QSk5ZjFxcXFrquPiqTW/oZmXv6gkt8sLee9TbHvZxMG9+SySYP4+PgizhjRmy55uSFXKcfDzJa6e3Hr9rQ5WCwiHaNi90Eee2sTzyzZyv7GCGMGdOcbl5/CJ08fwugB3cMuT1JAQSAiAGys3ce/v17G3OXbMeCqyUP49HmjKB7VV/v3M5yCQCTL7drfyL8tXM+Ti7ZQkJfD7eeP5nMXjmFYn65hlyYdREEgkqWiUeepxVv40atrOdAU4eZzRvDl6SdT1DO5P1aSzk9BIJKFtu48wD89v5xFG3dx4bgB3Hf1JMYP6hl2WRISBYFIlnm2ZBv3vVBKXo7xw+tP58biEToGkOUUBCJZorE5yndfLOWpxVs5/6T+/MtfncFQHQcQFAQiWaGmrp6/e3Ip72/dzd9eNJZvXH4Kebm6lIPEKAhEMtzWnQf49GOL2LG3kZ/deiZXTR4adknSySgIRDLYuuq93PboYhojUZ6+8zymjOgTdknSCSkIRDLUyoo9fOaxxeTl5vDMnR/jlME6K0japiAQyUCbduzns4+/R7eCPH79N+cyqr8uDSHt09EikQxTs7eezz6+GIAn71AIyNEpCEQyyN76Jm5/fAk79jbyX7efwxhdJE6OgYJAJENEo84Xn/4z66r38vPbzuIMHRiWY6QgEMkQD79Rxptra7nv6klcfMrAsMuRNJKUIDCzmWa21szKzOzuNpY/ZGbLgts6M9sdtywSt2xuMuoRyTZvr9/BTxasY9aUodx23qiwy5E0k/BZQ2aWCzwMXAaUA0vMbK67r2rp4+5fiev/ReDMuIc46O5TEq1DJFtV7jnIP875M+OKevB/rztd1w2S45aMLYKpQJm7b3T3RmAOMOsI/W8Bnk7C84pkvWjU+fKcZdQ3Rfj5bWfTrUBnhMvxS0YQDAO2xc2XB22HMbNRwBjg9bjmQjMrMbNFZnZte09iZncG/Upqa2uTULZI+nty8RYWb9rFfVdPYtzAHmGXI2mqow8W3ww85+6RuLZRwWDKtwI/NbOT2lrR3R9x92J3Ly4qKuqIWkU6tW27DvDgK2v4+PgB3Fg8IuxyJI0lIwgqgPi/wuFBW1tuptVuIXevCO43Am9y6PEDEWmDu3PPbz/AgAevn6zjApKQZATBEmC8mY0xswJiH/aHnf1jZhOAvsC7cW19zaxLMD0AuABY1XpdETnUnCXbeLtsB/dcOVFjC0vCEj6y5O7NZnYXMA/IBR5391Izux8ocfeWULgZmOPuHrf6ROCXZhYlFkoPxp9tJCKH+3B/Iw++sobzxvbj1qkjwy5HMkBSTjFw95eBl1u1fbvV/HfaWO8d4PRk1CCSLX66YB1765v47jWnkZOjXUKSOP2yWCSNrKvey5OLt3LruSN1WWlJGgWBSJpwd7730iq6FeTy1ctOCbscySAKApE08cbaGt5av4MvXTqeft0Lwi5HMoiCQCQNRKLOD15ew9gB3fnsx0aHXY5kGAWBSBp4acV2ymr28bUZp1CQp7etJJf+okQ6uUjU+deF6zllUE+uOG1w2OVIBlIQiHRyLy7fzsba/Xxp+nidLiopoSAQ6cQiUeffFq5nwuCezDxVWwOSGgoCkU5s7vIKNu7Yz5e1NSAppCAQ6aSiUeffXy9jwuCezJikrQFJHQWBSCf1+poaNtbu5+8vGaetAUkpBYFIJ/Xo2xsZ1qcrV+pMIUkxBYFIJ7SyYg+LNu7i9vNHk5ert6mklv7CRDqhx97eRPeCXG6aqpHHJPUUBCKdTOWeg7y4fDs3nTOSXoX5YZcjWSApQWBmM81srZmVmdndbSy/3cxqzWxZcLsjbtlsM1sf3GYnox6RdPbEO1uIuvPXF4wOuxTJEgkPTGNmucDDwGVAObDEzOa2MdLYM+5+V6t1+wH3AcWAA0uDdT9MtC6RdFTfFOHp97Yy87TBjOjXLexyJEskY4tgKlDm7hvdvRGYA8w6xnUvB15z913Bh/9rwMwk1CSSll5aUcmeg026wqh0qGQEwTBgW9x8edDW2vVmtsLMnjOzliNgx7quSFb49eItjC3qzrlj+oVdimSRjjpY/CIw2t0nE/vW/8TxPoCZ3WlmJWZWUltbm/QCRcK2pqqO97fu5tapIzHTD8ik4yQjCCqA+HPchgdtH3H3ne7eEMw+Cpx9rOvGPcYj7l7s7sVFRUVJKFukc/n14q0U5OVw/VnDwy5FskwygmAJMN7MxphZAXAzMDe+g5kNiZu9BlgdTM8DZphZXzPrC8wI2kSyyoHGZn73fgVXnjaYvhqGUjpYwmcNuXuzmd1F7AM8F3jc3UvN7H6gxN3nAv9oZtcAzcAu4PZg3V1m9j1iYQJwv7vvSrQmkXTz0vJK9jY0c+u5o8IuRbKQuXvYNRy34uJiLykpCbsMkaSZ9fD/sr+hmde+cpGOD0jKmNlSdy9u3a5fFouEbF31XpZv283N54xQCEgoFAQiIfvt+xXk5hjXnqkzpyUcCgKREEWizu//XMHFJxcxoEeXsMuRLKUgEAnROxt2UFVXz/Vn65RRCY+CQCREv32/gl6FeUybMDDsUiSLKQhEQrKvoZlXV1Zx1RlDKczPDbscyWIKApGQvLqyioNNEa4/SweJJVwKApGQ/Pb9ckb178ZZI/uGXYpkOQWBSAgq9xzk3Y07+dSZw/TbAQmdgkAkBH9YUYk7zJqi3UISPgWBSAheXFHJacN6MWZA97BLEVEQiHS0rTsPsHzbbq6aPDTsUkQABYFIh3vpg+0AfPL0IUfpKdIxFAQiHezF5ZWcObKPBqeXTkNBINKBymr2sbqyjqu1W0g6EQWBSAd6acV2zOCTk7VbSDqPpASBmc00s7VmVmZmd7ex/KtmtsrMVpjZQjMbFbcsYmbLgtvc1uuKZAp358Xl25k6uh+DehWGXY7IRxIOAjPLBR4GrgAmAbeY2aRW3f4MFLv7ZOA54Edxyw66+5Tgdk2i9Yh0Vuuq97Ghdj9XnaHdQtK5JGOLYCpQ5u4b3b0RmAPMiu/g7m+4+4FgdhGga+5K1plfWgXA5ZMGhVyJyKGSEQTDgG1x8+VBW3s+D7wSN19oZiVmtsjMrm1vJTO7M+hXUltbm1DBImGYv6qaM0f2YaB2C0kn06EHi83sNqAY+HFc86hgMOVbgZ+a2Ultrevuj7h7sbsXFxUVdUC1IsmzffdBPqjYw+WnDg67FJHDJCMIKoARcfPDg7ZDmNl04F7gGndvaGl394rgfiPwJnBmEmoS6VRadgvN0G4h6YSSEQRLgPFmNsbMCoCbgUPO/jGzM4FfEguBmrj2vmbWJZgeAFwArEpCTSKdyvxV1Ywb2IOxRT3CLkXkMAkHgbs3A3cB84DVwLPuXmpm95tZy1lAPwZ6AL9pdZroRKDEzJYDbwAPuruCQDLK7gONLN60i8tP1daAdE55yXgQd38ZeLlV27fjpqe3s947wOnJqEGks1q4uoZI1JkxSccHpHPSL4tFUmz+qioG9yrk9GG9wy5FpE0KApEUOtgY4Y/raplx6iBycjQSmXROCgKRFHprfS31TVHtFpJOTUEgkkLzV1XTqzCPc8f2C7sUkXYpCERSpDkSZeHqai6dOIj8XL3VpPPSX6dIiizZ/CEfHmjSj8ik01MQiKTI/FVVFOTlcNHJuiSKdG4KApEUcHfml1Zz0fgBdO+SlJ/riKSMgkAkBUq311Gx+6DOFpK0oCAQSYH5q6rJMbh04sCwSxE5KgWBSArML62ieHQ/+vfoEnYpIkelIBBJsq07D7Cmaq/OFpK0oSAQSbL5q4IhKTUIjaQJBYFIks0vrWbikF6M6Nct7FJEjomCQCSJduxrYMmWXdotJGlFQSCSRAtXV+MOMzQIjaSRpASBmc00s7VmVmZmd7exvIuZPRMsX2xmo+OW3RO0rzWzy5NRj0hY5pdWM7xvVyYN6RV2KSLHLOEgMLNc4GHgCmAScIuZTWrV7fPAh+4+DngI+GGw7iRiYxyfCswE/iN4PJG0s6+hmbfKdjBj0mDMNPaApI9kbBFMBcrcfaO7NwJzgFmt+swCngimnwMutdg7ZRYwx90b3H0TUBY8nkja+dO6Whqbo9otJGknGUEwDNgWN18etLXZJxjsfg/Q/xjXBcDM7jSzEjMrqa2tTULZIsk1v7SKft0LKB7VN+xSRI5L2hwsdvdH3L3Y3YuLinQ1R+lcGpujLFxTw6UTBpKnsQckzSTjL7YCGBE3Pzxoa7OPmeUBvYGdx7iuSKe3eNNO9tY3M0M/IpM0lIwgWAKMN7MxZlZA7ODv3FZ95gKzg+kbgNfd3YP2m4OzisYA44H3klCTSIeaX1pN1/xcPj5+QNiliBy3hC+U7u7NZnYXMA/IBR5391Izux8ocfe5wGPA/5hZGbCLWFgQ9HsWWAU0A//g7pFEaxLpSNGoM39VFZ84uYjCfJ30JuknKSNmuPvLwMut2r4dN10P/FU76z4APJCMOkTCsKJiD9V1DTpbSNKWjmqJJGh+aRW5OcalExQEkp4UBCIJmldaxXlj+9G7W37YpYicEAWBSALKavaxoXa/hqSUtKYgEEnAa6uqAbhMVxuVNKYgEEnAvNIqJg/vzdA+XcMuReSEKQhETlB1XT3Ltu3W2AOS9hQEIidofrBbSL8mlnSnIBA5QfNWVjF2QHfGD+wRdikiCVEQiJyA3QcaeXfjTmaeprEHJP0pCEROwILVNUSizszTtFtI0p+CQOQEvLqykqG9Czl9WO+wSxFJmIJA5Djta2jmT+t3cLl2C0mGUBCIHKc319bQ2BzlitOGhF2KSFIoCESO06srqxjQo4CzNSSlZAgFgchxqG+K8MaaGi6bNJjcHO0WksygIBA5Dm+v38H+xojOFpKMklAQmFk/M3vNzNYH94dtK5vZFDN718xKzWyFmd0Ut+xXZrbJzJYFtymJ1COSaq+WVtGrMI+Pje0fdikiSZPoFsHdwEJ3Hw8sDOZbOwB81t1PBWYCPzWzPnHLv+HuU4LbsgTrEUmZpkiUBaurmT5xEAV52piWzJHoX/Ms4Ilg+gng2tYd3H2du68PprcDNUBRgs8r0uHe27SL3QeauFy7hSTDJBoEg9y9MpiuAo54GUYzmwoUABvimh8Idhk9ZGZdjrDunWZWYmYltbW1CZYtcvxeXVlF1/xcLhqv7zGSWY4aBGa2wMxWtnGbFd/P3R3wIzzOEOB/gL9292jQfA8wATgH6Ad8s7313f0Rdy929+KiIr0RpWNFos680iouPqWIrgW5YZcjklR5R+vg7tPbW2Zm1WY2xN0rgw/6mnb69QL+ANzr7oviHrtla6LBzP4L+PpxVS/SQZZs3kXN3gauPF0/IpPMk+iuobnA7GB6NvBC6w5mVgD8Dvhvd3+u1bIhwb0RO76wMsF6RFLixeXb6Zqfy6UTB4ZdikjSJRoEDwKXmdl6YHowj5kVm9mjQZ8bgYuA29s4TfQpM/sA+AAYAHw/wXpEkq45EuWVlVVMnzSIbgVH3YgWSTsJ/VW7+07g0jbaS4A7gukngSfbWX9aIs8v0hHe2bCTXfsbuWqydgtJZtLJ0CJH8eLy7fTskscnTtZJCpKZFAQiR9DQHGFeaRWXnTqIwnydLSSZSUEgcgRvrdtBXX0zV58xNOxSRFJGQSByBC+u2E6fbvlcOG5A2KWIpIyCQKQdBxqbWbCqmpmnDiY/V28VyVz66xZpx/zSavY3RvjUmcPCLkUkpRQEIu14/v1yhvftyjmj+4VdikhKKQhE2lC1p57/LdvBdWcOI0cjkUmGUxCItOGFZRVEHT511vCwSxFJOQWBSCvuzvPvl3PWyD6MGdA97HJEUk5BINJK6fY61lXv4zptDUiWUBCItPL8++UU5OZw9WT9iEyyg4JAJE5TJMrcZduZPmkgvbvlh12OSIdQEIjEWbCqmp37G7nhbO0WkuyhIBCJ8+v3tjK0dyGfOFkD0Ej2SCgIzKyfmb1mZuuD+77t9IvEDUozN659jJktNrMyM3smGM1MJBRbdu7nrfU7uOmckeTqtwOSRRLdIrgbWOju44GFwXxbDrr7lOB2TVz7D4GH3H0c8CHw+QTrETlhT7+3jdwc46ZzRoRdikiHSjQIZgFPBNNPEBt3+JgE4xRPA1rGMT6u9UWSqbE5ynNLtzFtwkAG9y4MuxyRDpVoEAxy98pgugoY1E6/QjMrMbNFZnZt0NYf2O3uzcF8OaCre0ko5q+qYse+Rm49d2TYpYh0uKOOWWxmC4DBbSy6N37G3d3MvJ2HGeXuFWY2Fng9GLB+z/EUamZ3AncCjBypN6sk168Xb2VYn65cNF7DUUr2OWoQuPv09paZWbWZDXH3SjMbAtS08xgVwf1GM3sTOBN4HuhjZnnBVsFwoOIIdTwCPAJQXFzcXuCIHLeymn28s2EnX59xsg4SS1ZKdNfQXGB2MD0beKF1BzPra2ZdgukBwAXAKnd34A3ghiOtL5Jqj//vJgrycrh5qrY0JTslGgQPApeZ2XpgejCPmRWb2aNBn4lAiZktJ/bB/6C7rwqWfRP4qpmVETtm8FiC9Ygcl137G3l+aTnXnTmMAT26hF2OSCiOumvoSNx9J3BpG+0lwB3B9DvA6e2svxGYmkgNIol4ctEWGpqjfP7CMWGXIhIa/bJYslZ9U4T/fnczF59SxPhBPcMuRyQ0CgLJWnOXb2fHvkbuuHBs2KWIhEpBIFnJ3XnsrU1MGNyTC8b1D7sckVApCCQrvbG2hrXVe7nj42OJ/chdJHspCCTruDs/XbCeEf26MmuKBp8RURBI1nl9TQ0ryvfwxUvGk5+rt4CI3gWSVVq2Bkb268anztKlrURAQSBZZsHqGj6o2MNd08Zpa0AkoHeCZI3Y1sA6RvXvxnVnamtApIWCQLLGyx9UUbq9jrsuGUeetgZEPqJ3g2SF+qYIP3h5NRMG9+S6szQwvUg8BYFkhcfe3kTF7oN8+6pJutS0SCsKAsl4NXX1PPxGGTMmDeL8cQPCLkek01EQSMb70by1NEWi3PvJiWGXItIpKQgkoy3btpvnlpbzuQvGMKp/97DLEemUFASSsRqaI3zjN8sZ0ruQu6aNC7sckU4roSAws35m9pqZrQ/u+7bR5xIzWxZ3qzeza4NlvzKzTXHLpiRSj0i8n71exvqaffzgutPpWZgfdjkinVaiWwR3AwvdfTywMJg/hLu/4e5T3H0KMA04AMyP6/KNluXuvizBekQAWFmxh/94cwPXnzWcS04ZGHY5Ip1aokEwC3gimH4CuPYo/W8AXnH3Awk+r0i7GpujfOO5FfTrXsA/X6UDxCJHk2gQDHL3ymC6Chh0lP43A0+3anvAzFaY2UNm1u7o4WZ2p5mVmFlJbW1tAiVLpvt/89eyurKOB649jT7dCsIuR6TTO2oQmNkCM1vZxm1WfD93d8CP8DhDiA1iPy+u+R5gAnAO0A/4Znvru/sj7l7s7sVFRUVHK1uy1LzSKn75p418+tyRzDh1cNjliKSFvKN1cPfp7S0zs2ozG+LulcEHfc0RHupG4Hfu3hT32C1bEw1m9l/A14+xbpHDbNm5n6//ZjmTh/fm21dPCrsckbSR6K6hucDsYHo28MIR+t5Cq91CQXhgsbECrwVWJliPZKn6pghfePJ9csx4+Naz6JKXG3ZJImkj0SB4ELjMzNYD04N5zKzYzB5t6WRmo4ERwB9brf+UmX0AfAAMAL6fYD2ShaJR52vPLmdVZR0P3XQGI/p1C7skkbRy1F1DR+LuO4FL22gvAe6Im98MHHYBeHeflsjzi7g797+0ij98UMm9V05k2oSjna8gIq3pl8WS1n7+xw386p3N3HHhGP7morFhlyOSlhQEkraeXLSFH726lllThvKtK/V7AZETldCuIZGw/OKPG3jwlTVMmzCQH99wBjkaY0DkhCkIJK24O/8yfy0Pv7GBq88Yyk9uPEOD0IskSEEgaaO+KcK3X1jJsyXl3DJ1BN+/9nSNNiaSBAoCSQvbdx/kC08uZXn5Hv5x2ji+ctnJxH5+IiKJUhBIp/fW+lq+PGcZDc1RfvmZs7lcl44QSSoFgXRae+ub+MHLa3j6va2MG9iDX9x2NuMG9gi7LJGMoyCQTsfdeX1NDf/8+5VU1dXztxeN5SuXnUxhvi4bIZIKCgLpVFZW7OGBP6zm3Y07GTewB8994XzOGnnYwHcikkQKAukUVlbs4Rd/3MBLKyrp172A715zKreeO1Knhop0AAWBhKY5EuVP62t5/O3NvF22gx5d8vj7i0/i7y4+iV4aY1ikwygIpMOV1ezjt++X8/z75VTXNTCwZxfuuWICt5w7UgEgEgIFgaRcUyTKivI9LFxdzbzSKjbU7ifH4JJTBvLda4YzbcIgCvK0C0gkLAoCSbr6pgil2+tYtHEnizftomTzLg40RsjNMc4b24/Pfmw0V5w2mIG9CsMuVURQEEgCmiNRKnYfZPPOA6yprGN1ZR2rKuvYULufSDQ2fPUpg3pyw9nDOW9sf84/qb8GkxfphBIKAjP7K+A7wERgajAgTVv9ZgL/CuQCj7p7y0hmY4A5QH9gKfAZd29MpCZJjobmCLv2N1JT10B1XT01exuo2dtA7d56KnbXs3Xnfso/PEhz8IEPMLR3IROH9OLyUwdz6tBenDO6H/17dAnxVYjIsUh0i2AlcB3wy/Y6mFku8DBwGVAOLDGzue6+Cvgh8JC7zzGzXwCfB36eYE0Zw92JOjRHo0SiTnPUiUSC+6gT8Zb5vyxvjjgNzRHqm6LUN0Woj59uitDQ/Jfpg00R6g42U1ffRN3BJurqm6k72MSeg000NEcPq8cM+nfvwpDehZw2rDefnDyEUf27M6pfN04e1JO+3fVtXyQdJTpU5WrgaBf/mgqUufvGoO8cYJaZrQamAbcG/Z4gtnWRsiD41u8+YPHGnXz0HdbBiX3gtrS5g+O4/2WeI/SJbyN+vbh1DnuMQ9Zr1YfYh380+GBPlcL8HLrm59K7az69uubTqzCfwb0L6VUYm+/dNZ8+3fIZ1LOQgb26MLBnIQN6FJCn8/pFMk5HHCMYBmyLmy8HziW2O2i3uzfHtR82rnELM7sTuBNg5MiRJ1ZIn65MGNwLDFqiy8wwYt92W7e1NBj20XJr3RbXsd0+HBqWsTaL6xe0BQ0G5OUauTk55OUYuTl22P0hy3Ljl+XQJS+HwvxcCvNz6JIXuy/Mz6UwL5cu+bHlunKniLQ4ahCY2QKgrcs93uvuLyS/pLa5+yPAIwDFxcUn9FX5Hy4Zl9SaREQywVGDwN2nJ/gcFcCIuPnhQdtOoI+Z5QVbBS3tIiLSgTpih+8SYLyZjTGzAuBmYK67O/AGcEPQbzbQYVsYIiISk1AQmNmnzKwc+BjwBzObF7QPNbOXAYJv+3cB84DVwLPuXho8xDeBr5pZGbFjBo8lUo+IiBw/c0/dmSmpUlxc7CUlbf5kQURE2mFmS929uHW7zgUUEclyCgIRkSynIBARyXIKAhGRLJeWB4vNrBbYEnYdx2kAsCPsIjqYXnN20GtOH6Pcvah1Y1oGQToys5K2jtZnMr3m7KDXnP60a0hEJMspCEREspyCoOM8EnYBIdBrzg56zWlOxwhERLKctghERLKcgkBEJMspCEJgZl8zMzezAWHXkmpm9mMzW2NmK8zsd2bWJ+yaUsXMZprZWjMrM7O7w64n1cxshJm9YWarzKzUzL4Udk0dwcxyzezPZvZS2LUki4Kgg5nZCGAGsDXsWjrIa8Bp7j4ZWAfcE3I9KWFmucDDwBXAJOAWM5sUblUp1wx8zd0nAecB/5AFrxngS8QuqZ8xFAQd7yHgn/jL+PUZzd3nx41LvYjYSHSZaCpQ5u4b3b0RmAPMCrmmlHL3Snd/P5jeS+zDsd1xxzOBmQ0HPgk8GnYtyaQg6EBmNguocPflYdcSks8Br4RdRIoMA7bFzZeT4R+K8cxsNHAmsDjkUlLtp8S+yEVDriOpjjpmsRwfM1sADG5j0b3At4jtFsooR3rN7v5C0OdeYrsSnurI2iT1zKwH8DzwZXevC7ueVDGzq4Aad19qZheHXE5SKQiSzN2nt9VuZqcDY4DlZgaxXSTvm9lUd6/qwBKTrr3X3MLMbgeuAi71zP3hSgUwIm5+eNCW0cwsn1gIPOXuvw27nhS7ALjGzK4ECoFeZvaku98Wcl0J0w/KQmJmm4Fid0/HKxgeMzObCfwE+IS714ZdT6qYWR6xg+GXEguAJcCtceNzZxyLfaN5Atjl7l8OuZwOFWwRfN3drwq5lKTQMQJJtZ8BPYHXzGyZmf0i7IJSITggfhcwj9hB02czOQQCFwCfAaYF/7fLgm/Lkma0RSAikuW0RSAikuUUBCIiWU5BICKS5RQEIiJZTkEgIpLlFAQiIllOQSAikuX+P4LjS4iWHMEjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "x = torch.linspace(-5, 5, 200)\n",
    "y = torch.tanh(x)\n",
    "\n",
    "plt.plot(x.numpy(), y.numpy())\n",
    "plt.title(\"Tanh Function\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tanh 函数的优点是输出值域范围为 (-1,1)，适合二分类问题。缺点是当输入较大时，梯度较小，容易导致梯度消失问题。\n",
    "\n",
    "**3. ReLU 函数**\n",
    "\n",
    "ReLU 函数（Rectified Linear Unit）是一种线性函数，其表达式为：\n",
    "\n",
    "$$f(x) = \\max(0, x)$$\n",
    "\n",
    "运行下面的代码能够看到它的图像。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAEICAYAAAB25L6yAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAcHUlEQVR4nO3dd3zV9dnG8c9tZMhGCcgOCghUGSGCiFpErYqi1j4qAn0ea9WCUGf1cdTaaW1trVYRa32sVcIUB46qWFetM4O99yZhhxHIuJ8/cmhTDOQknHN+Z1zv1ysvkpMzrsO48uV7fuf+mbsjIiLx65igA4iIyJGpqEVE4pyKWkQkzqmoRUTinIpaRCTOqahFROKcilokwszsaTN7IOgckjxU1BIxZrbKzPaZ2W4z22Rmz5tZozBv+6GZ3XCY+zz/kMuuM7NPDnM/g8ysPJTh4MfrtXtGYeX+WhZ3H+Xuv4jWY0rqUVFLpA1190ZAb6APcG8AGTa4e6NKH0MDyCASMSpqiQp33wS8Q0VhA2BmZ5jZp2a2w8xmm9mgWOUJrbTXHXLZv1brZvZTM5tqZi+YWZGZzTezrErXbW9mL5tZoZltNbMnzaw78DQwILRy3xG67vNm9stKt73RzJaZ2TYzm2FmbSp9z81slJktDf2+jDMzi/JvhyQYFbVEhZm1Ay4GloW+bgu8CfwSOB74ETDdzNIDC/l1lwGTgWbADOBJADNLA94AVgMZQFtgsrsvBEYBn4VW7s0OvUMzGwz8GrgaaB26j8mHXO1S4HSgZ+h6F0b2aUmiU1FLpL1qZkXAWqAAeDB0+UjgLXd/y93L3X0mkAMMiUKGNqHV6cGPq8O83SehfGXAi0Cv0OX9gDbAXe6+x92L3b3KPfIqjACec/c8d99PxVbQADPLqHSdh919h7uvAT6g0v9CREBFLZF3hbs3BgYB3YAWocs7AldVLlDgLCpWmUdSCtQ55LI6QMkRbrPB3ZtV+pgaZvZNlT7fC9Q3s2OB9sBqdy8N834qa0PFKhoAd98NbKViVX64xw3rBVhJHSpqiQp3/wh4Hvhd6KK1wIuHFGhDd3+4mrtaQ8V2Q2WdqFR+YdoDNDj4RWg7I9xtl7VAh1BpH6q68ZMbqPghdfBxGwInAOvDfGwRFbVE1WPABWbWC5gADDWzC80szczqh17ga1fp+seGLj/4UQeYAtxmZt2sQhZwPV/f563OEipWyJeE7vfHQL0wb/slsBF42MwahrINDH1vM9DOzOoe5raTgO+ZWW8zqwc8BHzh7qtqmF9SmIpaosbdC4EXgJ+4+1rgcuA+oJCKVepd/OffwfHAvkoffwH+HPr1dWBn6P7ud/e3a5hlJ3Az8CwVq9k9wLoj3ujfty0DhgKdqVjhrwOuCX37fWA+sMnMtlRx2/eAB4DpVJT9ycCwmmQXMZ04QEQkvmlFLSIS51TUIiJxTkUtIhLnVNQiInGuquNCj1qLFi08IyMjGnctIpKUcnNzt7h7lcf2R6WoMzIyyMnJicZdi4gkJTM77Ju4tPUhIhLnVNQiInFORS0iEudU1CIicU5FLSIS58I66sPMVgFFQBlQ6u5ZR76FiIhESk0OzzvX3b82HUxERKJLWx8iIhHw5cptPPuPFURjImm4Re3Au2aWa2Y3VXUFM7vJzHLMLKewsDByCUVE4lxBUTFjJuaR/cUa9pWURfz+wy3qs9w9k4qzSo8xs3MOvYK7P+PuWe6elZ4eTyeWFhGJntKycn44MZ+i4hLGj8ykQd3Iv+E7rKJ29/WhXwuAV6g4K7OISMr73btL+GLlNh769ml0O7FJVB6j2qIOnSOu8cHPgW8B86KSRkQkgcxcsJmnP1rO8P4duDKzXfU3qKVw1uitgFfM7OD1J9b0fHUiIslm9dY93DF1Fqe1bcpPLu0R1ceqtqjdfQXQK6opREQSSHFJGaMn5HGMGU+NyKR+nbSoPl5UxpyKiCSzB1+bz4KNu3juuizaH98g6o+n46hFRGpg6ldrmZKzlrHndmZwt1YxeUwVtYhImOZv2MkDr81jYOcTuP2CrjF7XBW1iEgYdu4rYfSEPJo3qMvjw/qQdozF7LG1Ry0iUg1350fTZrNhxz6m/GAALRrVi+nja0UtIlKNP328gpkLNnPfkO707dg85o+vohYROYLPV2zlt28v4pKerfnewIxAMqioRUQOo2BXMWMn5pPRoiG/+U5PQm/8izntUYuIVKG0rJyxk/LZs7+UiTf2p1G94OpSRS0iUoVH3lnMlyu38dg1venaqnGgWbT1ISJyiHfmb+JPH69g5BkduKJP26DjqKhFRCpbtWUPP5o6m17tmvJAlIcthUtFLSISsu9AGaMm5JKWZowbkUm9Y6M7bClc2qMWEaHiTS0PvDaPxZuLeO6602nXPPrDlsKlFbWICDDlq7W8lLuOH57bmXNPaRl0nP+gohaRlDdv/U5+MmM+Z3dpwa3nx27YUrhU1CKS0nbuLWF0di4nNIz9sKVwaY9aRFJWeblz57RZbNpZzJQfDOD4hnWDjlQlrahFJGU9/fFy3ltYwP1DupPZIfbDlsKlohaRlPTp8i387p3FDO3Vhv85MyPoOEekohaRlLN5VzG3TMqnU4uGPHzlaYENWwqX9qhFJKWUlJUzdmIeew+UMenGM2gY4LClcMV/QhGRCPrt24v4atV2Hh/Wmy4BD1sKl7Y+RCRlvD1vI3/+x0r+e0BHLu8d/LClcKmoRSQlrCjczY+mzaFX+2bcf0n3oOPUiIpaRJLevgNl3JydR50046k4GrYULu1Ri0hSc3fuf3UuizcX8fz3+tG22XFBR6oxrahFJKlN+nItL+et55bBXfhm1/Sg49SKilpEktbcdTv56Yz5nNM1nVvO6xJ0nFpTUYtIUtqx9wCjs3Np0aguj13TOy6HLYVLe9QiknTKy507ps5m865ipo06M26HLYUr7BW1maWZWb6ZvRHNQCIiR2v8R8t5f1EBD1zag97tmwUd56jVZOvjVmBhtIKIiETCP5dt4ffvLuayXm347hkdg44TEWEVtZm1Ay4Bno1uHBGR2tu0s2LY0knpjfh1AgxbCle4K+rHgLuB8sNdwcxuMrMcM8spLCyMRDYRkbAdHLa0r6SMp0dmJsSwpXBVW9RmdilQ4O65R7qeuz/j7lnunpWenpjHKopI4nr4b4vIWb2d33ynJ51bJsawpXCFs6IeCFxmZquAycBgM5sQ1VQiIjXw5pyN/N8nK7nuzAyG9moTdJyIq7ao3f1ed2/n7hnAMOB9dx8Z9WQiImFYXribu1+aTZ8OzbhvSGINWwqX3vAiIglr74FSRk/IpV6dNMYNz6TusclZaTXabXf3D4EPo5JERKQG3J37X5nH0oLdvHB9P9ok4LClcCXnjx8RSXrZX6zhlfz13H5+V87uktwHMKioRSThzFm3g5+/voBBp6Qz9tzOQceJOhW1iCSU7XsOMHpCHumN6/GHq3tzTAIPWwpX8hwRLiJJr7zcuX3qLAqL9jNt1ACaJ/iwpXBpRS0iCWPcB8v4cHEhDwztQa8kGLYULhW1iCSET5Zu4dH3lnBF7zaM7N8h6DgxpaIWkbi3cec+bpmcT5eWjXgoiYYthUtFLSJx7UBpOWOy89hfUsb4kX1pUDf1XlpLvWcsIgnlobcWkrdmB+OGZ3JyeqOg4wRCK2oRiVuvz97A85+u4nsDM7ikZ+ug4wRGRS0icWlZwW7umT6HzA7NuPfi5By2FC4VtYjEnT37Kw1bGpG8w5bCldrPXkTijrtz3ytzWV64myeu7UPrpsk7bClcKmoRiSsTPl/Na7M2cMcFXRnYuUXQceKCilpE4sastTv4+RsLGNytJTcPSv5hS+FSUYtIXNi+5wBjsvNo1aQ+j17dKyWGLYVLx1GLSODKy53bplQMW3pp9ACaNUiNYUvh0opaRAL3xPvL+GhJIQ9e1oOe7ZoFHSfuqKhFJFAfLynksb8v4co+bRneL7WGLYVLRS0igVm/Yx+3Ts6na8vG/OrbqTdsKVwqahEJxMFhSyVlzviRmRxXNy3oSHFLLyaKSCB+9eYCZq3dwVMjMjkpRYcthUsrahGJuRmzN/DXz1bz/bM6MeS01B22FC4VtYjE1NLNRdwzfQ5ZHZtzz8Xdgo6TEFTUIhIze/aXMjo7jwZ103hyeCZ10lRB4dAetYjEhLtzz8tzWVG4mwk39OfEpvWDjpQw9ONMRGLihc9W8/rsDdz5rVM482QNW6oJFbWIRF3emu388s0FnNetJaO/eXLQcRKOilpEomrbngOMzc7jxKb1efTq3hq2VAvaoxaRqCkrd26dnM+WPQd4efSZNG1QJ+hICanaFbWZ1TezL81stpnNN7OfxSKYiCS+P/59Kf9YuoWfXfYNTm3bNOg4CSucFfV+YLC77zazOsAnZvY3d/88ytlEJIF9uLiAP76/lO9ktmPY6e2DjpPQqi1qd3dgd+jLOqEPj2YoEUls67bv5bYpszilVWN+ecWpGrZ0lMJ6MdHM0sxsFlAAzHT3L6q4zk1mlmNmOYWFhRGOKSKJYn9pGWOy8ygrc8aP7KthSxEQVlG7e5m79wbaAf3M7NQqrvOMu2e5e1Z6enqEY4pIovjlGwuZvW4nj1zVk04tGgYdJynU6PA8d98BfABcFJU0IpLQXpu1nhc/X82NZ3fiolM1bClSwjnqI93MmoU+Pw64AFgU5VwikmCWbC7inulz6ZdxPHdfpGFLkRTOUR+tgb+aWRoVxT7V3d+IbiwRSSS795cyakIuDesdy5PD+2jYUoSFc9THHKBPDLKISAJyd/53+hxWbdlD9g1n0LKJhi1Fmn7sichRef7TVbw5ZyN3XdiNASefEHScpKSiFpFay129nV+9uZDzu7di1DdPCjpO0lJRi0itbN29nzHZebRpdhy/v7qX3tQSRRrKJCI1Vlbu3DI5n217Q8OWjtOwpWjSilpEauyx95bwz2Vb+cXlGrYUCypqEamRDxYV8MT7y7iqbzuuOb1D0HFSgopaRMK2dlvFsKXurZvwiyu+NklCokRFLSJh2V9axpiJeZSXO+NHZFK/joYtxYpeTBSRsPz89QXMWbeTP323LxkathRTWlGLSLVeyV9H9hdr+ME5J3HhN04MOk7KUVGLyBEt3lTEvS/PpV+n47nrwlOCjpOSVNQiclhFxSWMnpBL4/p1eHJ4H47VsKVAaI9aRKp0cNjS6m17mXhDf1o21rCloOjHo4hU6bl/ruKtuZu4+8JT6H+Shi0FSUUtIl+Ts2obv35rId/q0YqbztGwpaCpqEXkP2zZvZ8xE/No2/w4HrlKw5bigfaoReRfysqdWybls2NvCa/c3E/DluKEilpE/uXRmYv5dPlWfvtfPenRpknQcSREWx8iAsDfF25m3AfLuSarPVdntQ86jlSiohYR1m7by+1TZtGjdRN+dvk3go4jh1BRi6S44pIyRmfn4sDTI/tq2FIc0h61SIr72esLmLd+F3/+7yw6nNAg6DhSBa2oRVLY9Nx1TPpyDaMHncwFPVoFHUcOQ0UtkqIWbdrF/a/OZcBJJ3DnBV2DjiNHoKIWSUG7iksYPSGPJvXr8MdrNWwp3mmPWiTFuDt3T5vDmm17mXTjGaQ3rhd0JKmGfoyKpJhn/7GSt+dv4p6LutGv0/FBx5EwqKhFUsiXK7fx8NuLuOgbJ3LD2Z2CjiNhUlGLpIiComLGTsyjffPj+O1VPTVsKYFoj1okBZSWlXPLpHx2FZfw1+v70aS+hi0lEhW1SAr4/cwlfL5iG7+7qhfdW2vYUqKpduvDzNqb2QdmtsDM5pvZrbEIJiKRMXPBZsZ/uJxr+7Xnv/q2CzqO1EI4K+pS4E53zzOzxkCumc109wVRziYiR2nN1r3cMXUWp7ZtwoNDNWwpUVW7onb3je6eF/q8CFgItI12MBE5OgeHLRkwfoSGLSWyGh31YWYZQB/giyq+d5OZ5ZhZTmFhYYTiiUht/XTGfOZv2MUfrulN++M1bCmRhV3UZtYImA7c5u67Dv2+uz/j7lnunpWenh7JjCJSQ9Ny1jL5q7WMOfdkzuuuYUuJLqyiNrM6VJR0tru/HN1IInI0FmzYxY9fnceZJ5/AHRecEnQciYBwjvow4P+Ahe7+aPQjiUht7dxXwujsXJo1qBi2lHaM3tSSDMJZUQ8EvgsMNrNZoY8hUc4lIjXk7tw1bTbrt+9j3PBMWjTSsKVkUe3hee7+CaAfyyJx7pmPV/Dugs38+JLuZGVo2FIy0awPkSTwxYqt/PadxQw57US+f5aGLSUbFbVIgivYVczYSfl0PL4Bv/mOhi0lI836EElgpWXljJ2UT1FxCS9+vx+NNWwpKamoRRLYI+8u5suV23j06l50O1HDlpKVtj5EEtS78zfxp49WMLx/B67M1LClZKaiFklAq7fu4c5pszmtbVN+cmmPoONIlKmoRRJMcUkZoybkcYwZT43I1LClFKA9apEE85PX5rFw4y7+ct3pGraUIrSiFkkgU75aw9ScdfxwcGfO7dYy6DgSIypqkQQxb/1OHnhtPmd1bsFt53cNOo7EkIpaJAHs3FfCzdl5HN+gLo8P661hSylGe9Qica683Llz6mw27NjHlB8M4AQNW0o5WlGLxLk/fbyC9xZu5r4h3enbsXnQcSQAKmqROPbZ8q088s4iLunZmu8NzAg6jgRERS0Spwp2FfPDSflktGioYUspTnvUInGopKycsRPz2bO/lIk39qdRPf1TTWX60xeJQ4+8s5gvV23jsWt607VV46DjSMC09SESZ96et4lnPl7ByDM6cEWftkHHkTigohaJIyu37OGuabPp1a4pD2jYkoSoqEXixL4DZYyekEtamjFuRCb1jtWwJamgPWqROODu/PjVeSzeXMRfrjudds01bEn+TStqkTgw+au1TM9bxw8Hd2HQKRq2JP9JRS0SsHnrd/LgjPmc3aUFt57XJeg4EodU1CIB2rm3hFETcjmhYV0eH9ZHw5akStqjFglIeblzx9RZbN5VzJQfDOD4hnWDjiRxSitqkYCM/2g5f19UwP1DupPZQcOW5PBU1CIB+HT5Fn7/7mKG9mrD/5yZEXQciXMqapEY27SzmFsm5dOpRUMevvI0DVuSammPWiSGKoYt5bH3QBmTbjyDhhq2JGHQ3xKRGPrN3xaRs3o7jw/rTRcNW5IwVbv1YWbPmVmBmc2LRSCRZPW3uRt59pOV/PeAjlzeW8OWJHzh7FE/D1wU5RwiSW1F4W7uemkOvds34/5LugcdRxJMtUXt7h8D22KQRSQp7T1QyugJedTRsCWppYgd9WFmN5lZjpnlFBYWRupuRRKau/PjV+axpKCIx4f1oW2z44KOJAkoYkXt7s+4e5a7Z6Wnp0fqbkUS2sQv1/By/npuPa8L53TVvwupHR1HLRIlc9bt4GczFnBO13RuGaxhS1J7KmqRKNix9wCjJ+TRolFdHrumN8do2JIchXAOz5sEfAacYmbrzOz70Y8lkrjKy53bp8yioKiYp0b21bAlOWrVvuHF3a+NRRCRZPHUh8v4YHEhP7/8G/Ru3yzoOJIEtPUhEkH/XLaFR2cu4bJebfjuGR2DjiNJQkUtEiEHhy2dlN6IX2vYkkSQZn2IREBJWTljJuaxr6SMKSMzNWxJIkp/m0Qi4NdvLSJ39XaeuLYPnVtq2JJElrY+RI7Sm3M28tw/V3LdmRkM7dUm6DiShFTUIkdhWcFu7n5pNpkdmnHfEA1bkuhQUYvU0t4DpdycnUu9OmmMG5FJ3WP1z0miQ3vUIrXg7tz38lyWFuzmhev70bqphi1J9GgJIFILE75Yw6uzNnD7+V05u4uGLUl0qahFamj22h384vUFDDolnbHndg46jqQAFbVIDWzfc4Cbs/NIb1yPP1ytYUsSG9qjFglTeblz+9RZFBbtZ9qoATTXsCWJEa2oRcL05AfL+HBxIQ8M7UEvDVuSGFJRi4ThH0sL+cN7S7iidxtG9u8QdBxJMSpqkWps2LGPWyfPokvLRjykYUsSABW1yBEcKK0YtrS/pIzxI/vSoK5e1pHY0986kSN46K2F5K/ZwbjhmZyc3ijoOJKitKIWOYwZszfw/KeruH5gJy7p2TroOJLCVNQiVVhWUMQ90+fQt2Nz7h3SLeg4kuJU1CKH2LO/lFET8jiuThrjhmdSJ03/TCRY2qMWqcTduffluawo3M2L3+/PiU3rBx1JRCtqkcpe/Hw1M2Zv4I4LujKwc4ug44gAKmqRf8lfs51fvLGAwd1acvMgDVuS+KGiFgG27TnAmOw8WjWpz6NX99KwJYkr2qOWlFdW7tw2ZRZbdh/gpdEDaNZAw5YkvmhFLSnvifeX8vGSQh68rAc92zULOo7I16ioJaV9tKSQx/++lCv7tGV4Pw1bkvikopaUtX7HPm6bnE/Xlo351bc1bEnil4paUtL+0jJuzs6jpMwZPzKT4+qmBR1J5LD0YqKkpF+9uZDZa3cwfkQmJ2nYksQ5ragl5bw2az0vfLaaG87qxMWnadiSxL+witrMLjKzxWa2zMzuiXYokWhZurmIe6bP5fSM5vzvxRq2JImh2qI2szRgHHAx0AO41sx6RDuYSKTN37CTG1/IoWG9NJ7UsCVJIOHsUfcDlrn7CgAzmwxcDiyIdJihT3xCcUlZpO9WBAdWbtlD8wZ1GT+yL62aaNiSJI5wirotsLbS1+uA/odeycxuAm4C6NChdsejnpzekANl5bW6rUh1BnVNZ+zgznrnoSSciB314e7PAM8AZGVleW3u47FhfSIVR0QkaYSzSbceaF/p63ahy0REJAbCKeqvgC5m1snM6gLDgBnRjSUiIgdVu/Xh7qVmNhZ4B0gDnnP3+VFPJiIiQJh71O7+FvBWlLOIiEgVdCCpiEicU1GLiMQ5FbWISJxTUYuIxDlzr9V7U458p2aFwOqI33F0tQC2BB0ixvScU4Oec2Lo6O7pVX0jKkWdiMwsx92zgs4RS3rOqUHPOfFp60NEJM6pqEVE4pyK+t+eCTpAAPScU4Oec4LTHrWISJzTilpEJM6pqEVE4pyKugpmdqeZuZm1CDpLtJnZI2a2yMzmmNkrZtYs6EzRkGonaDaz9mb2gZktMLP5ZnZr0JlixczSzCzfzN4IOkukqKgPYWbtgW8Ba4LOEiMzgVPdvSewBLg34DwRl6InaC4F7nT3HsAZwJgUeM4H3QosDDpEJKmov+4PwN1UnA816bn7u+5eGvrycyrO4JNs/nWCZnc/ABw8QXPScveN7p4X+ryIiuJqG2yq6DOzdsAlwLNBZ4kkFXUlZnY5sN7dZwedJSDXA38LOkQUVHWC5qQvrYPMLAPoA3wRcJRYeIyKhVZSnSU7Yie3TRRm9h5wYhXfuh+4j4ptj6RypOfs7q+FrnM/Ff9dzo5lNokuM2sETAduc/ddQeeJJjO7FChw91wzGxRwnIhKuaJ29/OrutzMTgM6AbPNDCq2APLMrJ+7b4phxIg73HM+yMyuAy4FzvPkPLA+JU/QbGZ1qCjpbHd/Oeg8MTAQuMzMhgD1gSZmNsHdRwac66jpDS+HYWargCx3T7QJXDViZhcBjwLfdPfCoPNEg5kdS8ULpedRUdBfAcOT+dyfVrHa+Cuwzd1vCzhOzIVW1D9y90sDjhIR2qOWJ4HGwEwzm2VmTwcdKNJCL5YePEHzQmBqMpd0yEDgu8Dg0J/rrNBKUxKQVtQiInFOK2oRkTinohYRiXMqahGROKeiFhGJcypqEZE4p6IWEYlzKmoRkTj3/1q+ZgWnA0vBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "x = torch.linspace(-5, 5, 200)\n",
    "y = torch.relu(x)\n",
    "\n",
    "plt.plot(x.numpy(), y.numpy())\n",
    "plt.title(\"ReLU Function\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ReLU 函数的优点是计算简单，速度快。缺点是当输入小于0时，输出为0，容易导致梯度消失问题。\n",
    "\n",
    "**4. Softmax 函数**\n",
    "\n",
    "Softmax 函数是一种常用的多分类激活函数，其表达式为：\n",
    "\n",
    "$$f_i(x) = \\frac{e^{x_i}}{\\sum_{j=1}^{n}e^{x_j}}$$\n",
    "\n",
    "其中，$n$ 是类别数，$x_i$ 是输入数据的第 $i$ 个元素，$f_i(x)$ 是输出数据的第 $i$ 个元素。\n",
    "\n",
    "Softmax 函数的作用是将输入数据转化为概率分布，使得每一类的概率和为1。它的优点是适合多分类问题，缺点是计算量较大。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhoklEQVR4nO3deXgd5Xn38e+t3ZJsybZkG0vCMtiG2GA2YSCFkgChJoE4fQsFskEDJQu0pVla0oWmvGn7Jk1D0hdaQgOUkAB2aJs6CdSQEkwJ4FjG2HjBRl4lb5JlWZtlbefuH2dkjoRsHdvnnDk6+n2uS5fmzDwzc498+afRMzPPmLsjIiKZKyvsAkREJLkU9CIiGU5BLyKS4RT0IiIZTkEvIpLhFPQiIhlOQS9py8x+28zqzazDzM4Lu550YmZ/ZmbfD7sOGR0U9JJUZnapmb1qZq1mdsDMfmVmF8a5+reAu9y9GGgxMzeznCSWGxczqw5q6Yj5WpPE/X3AzBpi57n737r77cnap2SW0P/TSOYyswnAz4DPA0uAPOAyoDvOTcwA1ienuoQodfe+sIsQGYnO6CWZ5gC4+1Pu3u/uXe7+vLuvBTCzLDP7CzPbYWaNZvYDMysxs3wz6wCygTVmtgV4OdjmweAM+hIzuzX4C+F+MztoZlvN7P3B/Ppgm7cMFGNmHzGz1WbWFiz/WsyyG81sW/DLCTO7xsz2mll5vAcbc6afEzPvJTO7PZi+1cxeMbNvmVlLsL9rYtpOMrPHzGx3sPwnZlYEPAdMj/nrYbqZfc3Mfhiz7kfNbH3wc3jJzN4Xs2y7mX3ZzNYGf1ktNrOCeI9LRj8FvSTTZqDfzB4PgnPikOW3Bl8fBE4DioEH3L076K4BOMfdTwd+M/hc6u7F7v5a8PkiYC0wGXgSeBq4EJgFfBJ4wMwGttUJfBooBT4CfN7MPgbg7ouBV4F/NLPJwCPA7e7elIgfRIyLgE1AGfBN4BEzs2DZE0AhMA+YAtzv7p3ANcDu4LiL3X137AbNbA7wFHA3UA48C/zUzPJimv0usBCYCcwn+nOXMUJBL0nj7m3ApYAD/wI0mdlSM5saNPkE8G133+ruHcBXgZuOsx9+m7s/5u79wGKgCrgv+GXxPNBDNPRx95fc/S13jwR/VTwFXB6zrTuBK4CXgJ+6+89G2Pf+4Az6oJl9Oc56d7j7vwT1Pg6cAkw1s1OIBvrn3L3F3XvdfXmc27wR+Lm7v+DuvUSvbYwD3h/T5h/dfbe7HwB+Cpwb57YlAyjoJancfaO73+rulcBZwHTgO8Hi6cCOmOY7iF43mkr89sVMdwX7HDqvGMDMLjKzX5pZk5m1Ap8jemY9UOtB4MdBnf8Qx77L3L00+PpWnPXujdnfoWCymOgvqAPu3hLndmIN+jm6ewSoByqG2y9wKNinjBEKekkZd38b+FeiQQqwm+gF1wGnAn0MDu8jqyeghCeBpUCVu5cADwED3SaY2bnAZ4ie6f/jCWy/M/heGDNvWpzr1gOTzKx0mGUjHfugn2PQFVQF7Ipz35LhFPSSNGZ2ppl9ycwqg89VwM3A60GTp4A/NrOZQT/63wKLj3InSxMQIdqXf6LGEz1rPmxmC4CPx9RaAPwQ+DPg94AKM/vC8Ww86M/fBXzSzLLN7DPA6XGuu4foRdd/MrOJZpZrZgPXJfYBk82s5CirLwE+YmZXmlku8CWidza9ejz1S+ZS0EsytRO9+LjCzDqJBvw6okEE8CjRC5AvA9uAw8AfDLehoJvjb4BfBX3iF59APV8A7jOzduBeogE54O+Aenf/Z3fvJnoh9+tmNvs49/H7wFeAZqIXVY8nbD8F9AJvA41EL64O/CX0FLA1OPbpsSu5+6ag3v8P7AeuA65z957jrF0ylOnFIyIimU1n9CIiGU5BLyKS4RT0IiIZTkEvIpLh0m5Qs7KyMq+urg67DBGRUWXVqlX73X3YsZnSLuirq6upra0NuwwRkVHFzHYcbZm6bkREMlxcQW9mC81sk5nVmdk9wyzPD4Y+rTOzFWZWHcyvNrMuM3sz+HoowfWLiMgIRuy6MbNs4EHgQ0ADsNLMlrr7hphmtwEt7j7LzG4CvkF0RD2ALe5+bmLLFhGReMVzRr8AqAuGku0hOt73oiFtFhEdchXgGeDKmDG2RUQkRPEEfQXRkfUGNDB4+NNBbYIBqVqJvggCYGbwVp/lZnbZcDswszvMrNbMapuaEv2eBxGRsS3ZF2P3AKe6+3nAF4EnB17VFsvdH3b3GnevKS+P+81tIiISh3iCfhfRsa0HVPLeca6PtAneDlQCNAdv+WkGcPdVwBaC94iKiEhqxBP0K4HZwZjhecBNRF/eEGspMPAS5uuBF93dzaw8uJiLmZ0GzAa2JqZ0EZHM8egr23j2rT1J2faIQR/0ud8FLAM2Akvcfb2Z3WdmHw2aPUL0xQh1RLtoBm7B/E1grZm9SfQi7eeCd1aKiEiMh5Zv4b83NiZl23E9GevuzxJ9s3zsvHtjpg8DNwyz3r8B/3aSNYqIZLS2w700tndz+pSipGxfT8aKiIRsa1P0dcOzypPzznYFvYhIyOoaOwA4fYqCXkQkI21p6iA32zh1UmFStq+gFxEJWV1jBzMmF5GbnZxIVtCLiIRsS1NH0vrnQUEvIhKq3v4IO5sPJe2OG1DQi4iEakdzJ30R53Sd0YuIZKa6xuDWyiTdcQMKehGRUL2zrx1AZ/QiIplqc2MHlRPHUZSfvFd4K+hFREK0eW87Z0wdn9R9KOhFRELS2x9h6/4OZivoRUQy0/b9nfT2O2dMS17/PCjoRURCs3lfdIyb2VN0Ri8ikpE27Wsny5J7ayUo6EVEQvPOvnZmTC6iIDc7qftR0IuIhGTTvnbmTE3u2Two6EVEQnG4t5/t+zuZk+Q7bkBBLyISiq1NnUQcBb2ISKbaHAx9cMY0Bb2ISEbavK+dnCyjenLyhiceoKAXEQnB5n3tnFZeRF5O8mNYQS8iEoLN+5I/9MEABb2ISIod6ulj54FDSR/MbICCXkQkxeoao0MfpOIeelDQi4ik3Ka90TtuUnFrJSjoRURSbuOedgpys5iRgjtuQEEvIpJyG/a0cua0CWRnWUr2p6AXEUkhd2fD7jbmTp+Qsn0q6EVEUmjXwS7aDvcx9xQFvYhIRtqwuw0g/c7ozWyhmW0yszozu2eY5flmtjhYvsLMqocsP9XMOszsywmqW0RkVFq/u40sg/dNS6OgN7Ns4EHgGmAucLOZzR3S7Dagxd1nAfcD3xiy/NvAcydfrojI6LZhTxszy4oYl5fcl43EiueMfgFQ5+5b3b0HeBpYNKTNIuDxYPoZ4EozMwAz+xiwDVifkIpFREax6IXYkpTuM56grwDqYz43BPOGbePufUArMNnMioE/Bf765EsVERndWg/1sutgV0ovxELyL8Z+Dbjf3TuO1cjM7jCzWjOrbWpqSnJJIiLh2LAn9RdiAXLiaLMLqIr5XBnMG65Ng5nlACVAM3ARcL2ZfRMoBSJmdtjdH4hd2d0fBh4GqKmp8RM4DhGRtLd+dytAys/o4wn6lcBsM5tJNNBvAj4+pM1S4BbgNeB64EV3d+CygQZm9jWgY2jIi4iMFRv2tDFlfD7l4/NTut8Rg97d+8zsLmAZkA086u7rzew+oNbdlwKPAE+YWR1wgOgvAxERiZHqJ2IHxHNGj7s/Czw7ZN69MdOHgRtG2MbXTqA+EZGM0N3XT11jB1ecOSXl+9aTsSIiKfDOvg76Ih7KGb2CXkQkBY4MfZDiC7GgoBcRSYl1u1spysumOkVj0MdS0IuIpMCa+oOcXVlCVorGoI+loBcRSbKevggb97RzTmVpKPtX0IuIJNnbe9vo6Y8wX0EvIpKZ1jREn4idX5nawcwGKOhFRJJsTf1BJhflUTlxXCj7V9CLiCTZ2oaDzK8sIRi9PeUU9CIiSdTZ3UddY0do/fOgoBcRSap1u1qJOJxTFU7/PCjoRUSSak3DQQCd0YuIZKo1Da1UlI6jrDi1QxPHUtCLiCTR2oaDoXbbgIJeRCRpDnT2UH+gK7QnYgco6EVEkiQd+udBQS8ikjRr61sxg7NDeiJ2gIJeRCRJVu1sYc6U8RTnx/Uyv6RR0IuIJEF/xFm9o4ULqieGXYqCXkQkGd5pbKe9u4+aGQp6EZGMVLu9BYCaGZNCrkRBLyKSFKt2tFBWnE/VpHBGrIyloBcRSYLaHQeomTExtBErYynoRUQSrLHtMPUHuqhJgwuxoKAXEUm42h3R/vkL0uBCLCjoRUQSrnZ7C/k5WcybHu6DUgMU9CIiCbZqxwHOqSolLyc9IjY9qhARyRBdPf2s392WFvfPD1DQi4gk0Jv1B+mLeNpciAUFvYhIQr2xM3oh9vxTFfQiIhnp9a3NzJ5STGlhXtilHKGgFxFJkJ6+CLXbW3j/6ZPDLmWQuILezBaa2SYzqzOze4ZZnm9mi4PlK8ysOpi/wMzeDL7WmNlvJ7h+EZG0sbbhIF29/Vwy2oLezLKBB4FrgLnAzWY2d0iz24AWd58F3A98I5i/Dqhx93OBhcD3zCzcgZlFRJLktS3NmMFFM0dZ0AMLgDp33+ruPcDTwKIhbRYBjwfTzwBXmpm5+yF37wvmFwCeiKJFRNLRa1ubOXPaBCYWpU//PMQX9BVAfcznhmDesG2CYG8FJgOY2UVmth54C/hcTPAfYWZ3mFmtmdU2NTUd/1GIiISsu6+fVTtauOS09DqbhxRcjHX3Fe4+D7gQ+KqZFQzT5mF3r3H3mvLy8mSXJCKScKt3HqS7L5J2/fMQX9DvAqpiPlcG84ZtE/TBlwDNsQ3cfSPQAZx1osWKiKSr17Y0k2WwYGb4LxoZKp6gXwnMNrOZZpYH3AQsHdJmKXBLMH098KK7e7BODoCZzQDOBLYnpHIRkTTy2pZm5k0voWRcbtilvMeIQR/0qd8FLAM2Akvcfb2Z3WdmHw2aPQJMNrM64IvAwC2YlwJrzOxN4D+AL7j7/gQfg4hIqLp6+lld35KW3TYAcd3q6O7PAs8OmXdvzPRh4IZh1nsCeOIkaxQRSWurdrTQ2+9peSEW9GSsiMhJe23rfrKzjAvTsH8eFPQiIiftlbpmzqksoTg/PZ8HVdCLiJyEA509rG04yOVzpoRdylEp6EVETsL/vNOEO1x+Rvo+A6SgFxE5Ccs3NzGxMJezK9Lj/bDDUdCLiJygSMR5efN+LptdTnaWhV3OUSnoRURO0IY9bezv6ObyOenbbQMKehGRE7Z8c3QQxsvmlIVcybEp6EVETtDyzU3Mmz6BKePfM1ZjWlHQi4icgLbDvbyxoyXtu21AQS8ickJerWumL+IKehGRTLV8cxPj83M4f8bEsEsZkYJeROQ4RSLOi2/v49LZZeRmp3+Mpn+FIiJpZu2uVva1dXP1vKlhlxIXBb2IyHF6YcNesrOMD56RvuPbxFLQi4gcp+fX7+OimZMoLcwLu5S4KOhFRI7Dtv2dvNPYwdVzR0e3DSjoRUSOywsb9gJwlYJeRCQzPb9+H/OmT6ByYmHYpcRNQS8iEqf9Hd2s2tnC1XOnhV3KcVHQi4jE6b837sMdPjSKum1AQS8iErfn1++jcuI43nfK+LBLOS4KehGROLQe6uXld5pYOG8aZun7kpHhKOhFROKwbMNeevudj547PexSjpuCXkQkDj9ds5sZkwvT+t2wR6OgFxEZwf6Obl7d0sx186ePum4bUNCLiIzoubf20B9xrjtn9HXbgIJeRGREP12zhzlTizlj2ui622aAgl5E5Bj2tHbx6+0HuG7+6DybBwW9iMgx/XztHgCuHaXdNqCgFxE5pqVrdnN2RQkzy4rCLuWExRX0ZrbQzDaZWZ2Z3TPM8nwzWxwsX2Fm1cH8D5nZKjN7K/h+RYLrFxFJmk1721nb0MrHzqsIu5STMmLQm1k28CBwDTAXuNnM5g5pdhvQ4u6zgPuBbwTz9wPXufvZwC3AE4kqXEQk2X5cW09OlvGxUfiQVKx4zugXAHXuvtXde4CngUVD2iwCHg+mnwGuNDNz99XuvjuYvx4YZ2b5iShcRCSZevsj/OTNXVz1vqlMLh7dsRVP0FcA9TGfG4J5w7Zx9z6gFZg8pM3vAG+4e/fQHZjZHWZWa2a1TU1N8dYuIpI0v3y7kf0dPdxQUxl2KSctJRdjzWwe0e6czw633N0fdvcad68pLy9PRUkiIsf041UNlI/P5/I5oz+T4gn6XUBVzOfKYN6wbcwsBygBmoPPlcB/AJ929y0nW7CISLI1tXfz4tuN/J/zK8jJHv03J8ZzBCuB2WY208zygJuApUPaLCV6sRXgeuBFd3czKwV+Dtzj7r9KUM0iIkn1k9W76I84N1xQNXLjUWDEoA/63O8ClgEbgSXuvt7M7jOzjwbNHgEmm1kd8EVg4BbMu4BZwL1m9mbwNSXhRyEikiDuzpLaes47tZRZU4rDLichcuJp5O7PAs8OmXdvzPRh4IZh1vs68PWTrFFEJGVWbDvAO40dfPN35oddSsKM/s4nEZEE+sFr2yktzB2VLxg5GgW9iEhgb+thlq3fx401VRTkZoddTsIo6EVEAk+u2EHEnU9ePCPsUhJKQS8iAnT39fPkr3dyxRlTqJpUGHY5CaWgFxEB/mvdXvZ39PDp91eHXUrCKehFRIDHX93OzLIiLptVFnYpCaegF5Exb9WOFt7YeZBPXzKDrKzR9/LvkSjoRWTMe2j5FkoLc7nxwsx4EnYoBb2IjGl1je28sGEfn76kmsK8uJ4hHXUU9CIypn1v+VYKcrO4NQMvwg5Q0IvImLWntYufvLmLG2uqmFSUF3Y5SaOgF5Ex69FXthFxuP2y08IuJakU9CIyJrV09vDkip1cO/+UjHtAaigFvYiMSd97eSuHevu584Ozwi4l6RT0IjLmNLV38/ir21l0znTmTB0fdjlJp6AXkTHnn1/aQk9/hD+6ak7YpaSEgl5ExpQ9rV38cMUOfuf8CmaWFYVdTkoo6EVkTHnwl3W4O39wxeywS0kZBb2IjBnb93eyeGU9N15YlfF32sRS0IvImPF3z20kLzuLPxxDZ/OgoBeRMeLVLftZtn4fX/jgLKZMKAi7nJRS0ItIxuuPOF//2UYqSsdx26Uzwy4n5RT0IpLxnllVz4Y9bdxzzZkZ9dLveCnoRSSjtR/u5e+XbeaCGRO5dv4pYZcTCgW9iGS0by3bRHNnN3913VzMMu/tUfFQ0ItIxlq9s4UfvL6DWy6pZn5ladjlhEZBLyIZqbc/wlf//S2mji/gS1ePjaEOjiYz35slImPe9/9nG2/vbed7n7qA8QW5YZcTKp3Ri0jG2b6/k+/+92aunjuV35o3LexyQqegF5GM0tcf4Y+XvEledhZ/vWhe2OWkBXXdiEhGeWj5FlbvPMh3bzqXU0rGhV1OWojrjN7MFprZJjOrM7N7hlmeb2aLg+UrzKw6mD/ZzH5pZh1m9kCCaxcRGeSthla+84t3uO6c6Sw6tyLsctLGiEFvZtnAg8A1wFzgZjObO6TZbUCLu88C7ge+Ecw/DPwl8OWEVSwiMozDvf3cvXg1ZcX5fH3RWWGXk1biOaNfANS5+1Z37wGeBhYNabMIeDyYfga40szM3Tvd/RWigS8ikjT3/uc6tjR18q0bzqGkcGzfZTNUPEFfAdTHfG4I5g3bxt37gFZgciIKFBEZyZKV9SypbeAPr5jFpbPLwi4n7aTFXTdmdoeZ1ZpZbVNTU9jliMgosn53K3/5n+u4dFbZmHkH7PGKJ+h3AVUxnyuDecO2MbMcoARojrcId3/Y3Wvcvaa8vDze1URkjGvt6uULP3qDiYV5fPemc8nOGptj2YwknqBfCcw2s5lmlgfcBCwd0mYpcEswfT3wort74soUERmstz/CnT96g10tXTzw8fOYXJwfdklpa8T76N29z8zuApYB2cCj7r7ezO4Dat19KfAI8ISZ1QEHiP4yAMDMtgMTgDwz+xhwtbtvSPiRiMiY4e781dL1vFK3n29eP5+a6klhl5TW4npgyt2fBZ4dMu/emOnDwA1HWbf6JOoTEXmPR17ZxpMrdvL5D5zO79ZUjbzCGJcWF2NFROL1X+v28DfPbuSas6bxlavPCLucUUFBLyKjxsubm/iDp1ZzXlUp3/7dc8nSxde4KOhFZFSo3X6AO56oZdaU8Tz2ewsYlzf23v16ohT0IpL21u1q5ff+dSXTS8bxg88soGScnnw9Hgp6EUlrb+xs4eZ/eZ0JBbk8cftFlI/XbZTHS0EvImnr9a3NfOr7K5hUlMeSz11CRamGHT4RCnoRSUu/3NTIrY/9mmklBSz5rEL+ZCjoRSTtPLliJ7c/XstpZcUs/uwlTJ1QEHZJo5reMCUiaSMScb71/Cb+6aUtXD6nnAc/cT7F+Yqpk6WfoIikhY7uPr7y4zU8t24vNy84lf+7aB452ep0SAQFvYiEbktTB599YhVbmzr48w+/j9svm4mZHoZKFAW9iITqubf28JVn1pKXk8UPb7uI98/Si0MSTUEvIqHo7O7jvp9uYHFtPedUlfLPnzif6bqzJikU9CKScqt3tvDFJWvY3tzJnR88nbuvmkOu+uOTRkEvIilzqKePf3h+M4/9ahvTJhTw1O9fzMWn6fXSyaagF5GUeGlTI3/xk3U0tHTxyYtP5U8WnsmEAo1ZkwoKehFJqu37O/n6zzfwi42NnFZWxOI7LuYincWnlIJeRJKipbOHh5Zv4bFfbSc32/jThWfymUuryc/R8MKppqAXkYRq7erlkVe28egr2+js6eO3z6vgTxeeqWEMQqSgF5GEaDvcyxOv7eB7y7fQdriPD589jbuvmsOcqePDLm3MU9CLyEnZ2XyIx17dxo9rG+jo7uOq903h7qvmcFZFSdilSUBBLyLHzd1Zse0Aj/1qGy9s2EeWGdfOP4XbLj2NsysV8OlGQS8icdvT2sW/v7GLH9fWs735EKWFuXzu8tP59CXVTCtRH3y6UtCLyDG1He7lxY2N/OTNXby8uYmIw4KZk7jritl8+OxpFOYpRtKd/oVE5D0OHurhhQ37eG7dXl55Zz89/RFOKSngzg/O4voLKpkxuSjsEuU4KOhFhEjEWbe7leWbmli+uYnV9QfpjzgVpeO45f0zWHjWKZxXVUpWloYOHo0U9CJjUH/E2bS3ndodB/j1tgO8tqWZ5s4eAOZXlvD5y0/nt+ZN46yKCRoXPgMo6EXGgNauXjbsbuONnS2s3H6AVTtaaD/cB8C0CQVcNruMD5wxhUtnl1FWnB9ytZJoCnqRDOLuNLV3s2FPG+t3t7F+dyvrdrWx88ChI21mTynmunOmc2H1RC6snkRF6TidtWc4Bb3IKHS4t59dB7vY0tjBlqZOtjR1UNfYwZamjiNn6gAzJhdydkUJN15YxVkVJcyvKGFiUV6IlUsYFPQiacbdaTvcR1N7N3tau2ho6aKh5RD1B6LfG1q6aGzvHrTOlPH5zJpSzMfOreD08iLOPGUCc6dP0DDAAijoRVKipy9Ca1cvrV09HDzUS8uhXpo7umls76apvZvG9sPB9+jn7r7IoPWzs4zppQVUTSzkA2eUUzmxkMqJ4zitvJjTyosU6HJMcQW9mS0EvgtkA9939/83ZHk+8APgAqAZuNHdtwfLvgrcBvQDf+juyxJWvUgSuTvdfRG6eyN09PTR2T3w1U/HwHRP9HNnd9+gea1dvRw8FP1q7eqlo7vvqPspLcylvDifKRPyqZkxkSkTCo58njqhgKpJhUwdn0+OXrUnJ2jEoDezbOBB4ENAA7DSzJa6+4aYZrcBLe4+y8xuAr4B3Ghmc4GbgHnAdOAXZjbH3fsTfSCSfO5OxCHiTsQdPzId/e6Rd5dF/Ojt+yPHXh7x6O1/ff0R+iNOb8Tpj0To7Y+u29sfoW9gOhK06X+3Td/AdOTd9r390cDu7ovQ3ddPd1+Ew739R4L83XnvTvcMOas+liyDovwcivNzKMzLprQwj2kTCjhj2nhKx+VRWpjLxMJcSgrzKB2XS8m4XMrG51NWnKfx2SXp4jmjXwDUuftWADN7GlgExAb9IuBrwfQzwAMWvYy/CHja3buBbWZWF2zvtcSU/66397Zx15OrgWiAAHhsAx/07UibwfNim/t75w3a4Alu4z3tYrbhQ+e891gG1zPycY64jWPUiEeXHwnyIcef7nKzjZysLHKyjJxsoyA3m4LcbPJzsoKvbIrzc5hclEX+kfnB99zodEHwvSgv+0iQF+XnUJSfTVFezpF5BblZunNF0lY8QV8B1Md8bgAuOlobd+8zs1ZgcjD/9SHrVgzdgZndAdwBcOqpp8Zb+yAFOdmcETvutQ36NrCfQfNi/1/akDaD1h3UzgatO3j7g9sMmjcoA05wG8PUOLTNcDUOXve92zjWcWYZZJmRZdF1BqazsmKmzbCYdllZFrQ9yrpB++yYbQy3PDc7i+wsOxLY0el35w18zskKAj1m3sA2RSRNLsa6+8PAwwA1NTUndN5YXVbEg584P6F1iYhkgniu7uwCqmI+Vwbzhm1jZjlACdGLsvGsKyIiSRRP0K8EZpvZTDPLI3pxdemQNkuBW4Lp64EXPdoxvBS4yczyzWwmMBv4dWJKFxGReIzYdRP0ud8FLCN6e+Wj7r7ezO4Dat19KfAI8ERwsfUA0V8GBO2WEL1w2wfcqTtuRERSyzzNbqWoqanx2trasMsQERlVzGyVu9cMt0xPYIiIZDgFvYhIhlPQi4hkOAW9iEiGS7uLsWbWBOwIu44TUAbsD7uIFNMxjw1j7ZhH6/HOcPfy4RakXdCPVmZWe7Qr3plKxzw2jLVjzsTjVdeNiEiGU9CLiGQ4BX3iPBx2ASHQMY8NY+2YM+541UcvIpLhdEYvIpLhFPQiIhlOQZ8EZvYlM3MzKwu7lmQzs783s7fNbK2Z/YeZlYZdUzKY2UIz22RmdWZ2T9j1JJuZVZnZL81sg5mtN7M/CrumVDGzbDNbbWY/C7uWRFHQJ5iZVQFXAzvDriVFXgDOcvf5wGbgqyHXk3Bmlg08CFwDzAVuDl58n8n6gC+5+1zgYuDOMXDMA/4I2Bh2EYmkoE+8+4E/Ycg7uzOVuz/v7n3Bx9eJvkUs0ywA6tx9q7v3AE8TffF9xnL3Pe7+RjDdTjT43vO+50xjZpXAR4Dvh11LIinoE8jMFgG73H1N2LWE5DPAc2EXkQQVQH3M52Ffcp+pzKwaOA9YEXIpqfAdoidqkZDrSKi0eDn4aGJmvwCmDbPoz4E/I9ptk1GOdczu/p9Bmz8n+uf+j1JZmySXmRUD/wbc7e5tYdeTTGZ2LdDo7qvM7AMhl5NQCvrj5O5XDTffzM4GZgJrzAyiXRhvmNkCd9+bwhIT7mjHPMDMbgWuBa70zHwwY0y+5N7McomG/I/c/d/DricFfgP4qJl9GCgAJpjZD939kyHXddL0wFSSmNl2oMbdR+MoeHEzs4XAt4HL3b0p7HqSwcxyiF5ovpJowK8EPu7u60MtLIkserbyOHDA3e8OuZyUC87ov+zu14ZcSkKoj15O1gPAeOAFM3vTzB4Ku6BECy423wUsI3pRckkmh3zgN4BPAVcE/65vBme6MgrpjF5EJMPpjF5EJMMp6EVEMpyCXkQkwynoRUQynIJeRCTDKehFRDKcgl5EJMP9L8clI0sYSWd+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "x = torch.linspace(-5, 5, 200)\n",
    "y = torch.softmax(x, dim=0)\n",
    "\n",
    "plt.plot(x.numpy(), y.numpy())\n",
    "plt.title(\"Softmax Function\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**梗直哥提示：其实介绍了这么多激活函数，你只要记着Sigmoid和Tanh 函数适用于二分类问题，ReLU 函数适用于大多数任务，Softmax函数适用于多分类问题就行了。当然实战中，还有很多各种各样的变体。如果你对这方面的进阶内容和实战内容感兴趣，欢迎入群学习（加V: gengzhige99）**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Next 5-2多层感知机代码实现](./5-2%20多层感知机代码实现.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
